integrated image reshaping and video encoding

专利摘要:
given a sequence of images in a first code word representation, methods, processes and systems are presented for integrating remodeling into a next generation video codec to encode and decode the images, in which the remodeling allows part of the images to be encoded in a second codeword representation that allows for more efficient compression than using the first codeword representation. A variety of architectures are discussed, including: an out-of-loop remodeling architecture, an intra-loop only remodeling architecture, a loop architecture for prediction residuals, and a hybrid loop remodeling architecture. there are also syntax methods for signaling remodeling parameters, and image coding methods optimized in relation to remodeling.
公开号:BR112019016885A2
申请号:R112019016885
申请日:2018-06-29
公开日:2020-04-14
发明作者:Pu Fangjun；Yin Peng；Chen Tao；Lu Taoran；J Husak Walter
申请人:Dolby Laboratories Licensing Corp；
IPC主号:

专利说明:

“INTEGRATED IMAGE REMODELING AND VIDEO ENCODING” REFERENCE TO THE RELATED DEPOSIT APPLICATIONS [001] This application claims priority to the Provisional Patent Application with U.S. Serial Number 62 / 686,738, filed on June 19, 2018; to Provisional Patent Application with U.S. Serial Number 62 / 680,710, filed on June 5, 2018; Provisional Patent Application with U.S. Serial Number 62 / 629,313, filed on February 12, 2018; Provisional Patent Application with U.S. Serial Number 62 / 561,561, filed on September 21, 2017; and to Provisional Patent Application with U.S. Serial Number 62 / 526,577, filed on June 29, 2017, each of which is incorporated into this document in its entirety for reference.
TECHNOLOGY [002] The present invention relates, in general, to images and video coding. More particularly, one embodiment of the invention relates to integrated image reshaping and video encoding.
FUNDAMENTALS [003] In 2013, the MPEG group at the International Organization for Standardization (ISO), together with the International Telecommunications Union (ITU), published the first draft of the HEVC video encoding standard (also known as H.265). More recently, the same group issued an invitation to support the development of a next generation coding standard that provides enhanced coding performance over existing video encoding technologies.
[004] Depending on the usage in question, the term 'bit depth' denotes the number of pixels used to represent one of the color components of an image. Traditionally, images have been encoded in 8 bits, per color component, per pixel (for example, 24 bits per pixel); however, modern architectures
Petition 870200044274, of 07/04/2020, p. 8/88
2/76 can now support higher bit depths, such as 10 bits, 12 bits or more.
[005] In a traditional image pipeline, captured images are quantized using a non-linear optoelectronic function (OETF), which converts linear scene light into a non-linear video signal (for example, gamma-encoded RGB or YCbCr). Then, at the receiver, before being displayed on the screen, the signal is processed by an electro-optical transfer function (EOTF) that translates the video signal values into output screen color values. These non-linear functions include the traditional “gamma” curve, documented in ITU-R Rec. BT.709 and BT. 2020, and the “PQ” curve (perceptual quantization), described in SMPTE ST 2084 and Rec. ITU-R BT. 2100.
[006] Depending on the usage in question, the term “progressive remodeling” denotes a process of mapping sample to sample or code word to code word of a digital image from its original bit depth and word distribution or representation -original code (eg gamma or PQ, and the like) to an image with the same or different bit depth and a different distribution or representation of code words. Remodeling allows for improved compression capacity or improved image quality at a fixed bit rate. For example, without limitation, remodeling can be applied to 10-bit or 12-bit PQ-encoded HDR video to improve encoding efficiency in a 10-bit video encoding architecture. In a receiver, after unpacking the refurbished signal, the receiver can apply a “reverse remodeling function” to re-store the signal to its original code word distribution. As assessed by the inventors, as development begins for the next generation of a video encoding standard, improved techniques for integrated image reshaping and encoding are desired. The methods of this invention may be applicable to a variety of video content, including, without limitation, in-band content
Petition 870200044274, of 07/04/2020, p. 9/88
3/76 standard dynamics (SDR) and / or high dynamic range (HDR).
[007] The approaches described in this section are approaches that can be acquired, but not necessarily approaches that were previously conceived or acquired. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, it should not be assumed that the issues identified in relation to one or more approaches have been recognized in any prior art based on this section, except where indicated otherwise.
BRIEF DESCRIPTION OF THE DRAWINGS [008] One embodiment of the present invention is illustrated by way of example, and without limitation, in the figures in the accompanying drawings and in which similar numerical references refer to similar elements, in which:
[009] Figure 1A depicts an example process for a video deployment pipeline;
[010] Figure 1B depicts an exemplary process for data compression using signal reshaping according to the prior art;
[011] Figure 2A depicts an exemplary architecture for an encoder using normative loop remodeling according to one embodiment of this invention;
[012] Figure 2B depicts an exemplary architecture for a decoder using normative loop remodeling according to one embodiment of this invention;
[013] Figure 2C depicts an exemplary architecture for an encoder using normative loop remodeling only intra according to one embodiment of this invention;
[014] Figure 2D depicts an exemplary architecture for a
Petition 870200044274, of 07/04/2020, p. 10/88
4/76 decoder using normative loop remodeling only intra according to one embodiment of this invention;
[015] Figure 2E depicts an exemplary architecture for an encoder using loop remodeling for prediction residuals according to one embodiment of this invention;
[016] Figure 2F depicts an exemplary architecture for a decoder using loop remodeling for prediction residuals according to one embodiment of this invention;
[017] Figure 2G depicts an exemplary architecture for an encoder using hybrid loop remodeling according to one embodiment of this invention;
[018] Figure 2H depicts an exemplary architecture for a decoder using hybrid loop remodeling according to one embodiment of this invention;
[019] Figure 3A depicts an exemplary process for encoding video using an out-of-loop remodeling architecture according to an embodiment of this invention;
[020] Figure 3B depicts an exemplary process for video decoding using an out-of-loop remodeling architecture according to one embodiment of this invention;
[021] Figure 3C depicts an exemplary process for encoding video using a loop-only remodeling architecture according to one embodiment of this invention;
[022] Figure 3D depicts an exemplary process for video decoding using a loop-only remodeling architecture according to one embodiment of this invention;
[023] Figure 3E depicts an exemplary process for coding
Petition 870200044274, of 07/04/2020, p. 11/88
5/76 video using a loop remodeling architecture for prediction residuals according to an embodiment of this invention;
[024] Figure 3F depicts an exemplary process for video decoding using a loop remodeling architecture for prediction residuals according to one embodiment of this invention;
[025] Figure 4A depicts an exemplary process for encoding video using any of, or a combination of, three architectures based on remodeling according to one embodiment of this invention;
[026] Figure 4B depicts an exemplary process for video decoding using any of, or a combination of, three architectures based on remodeling according to one embodiment of this invention;
[027] Figures 5A and 5B depict a process of reconstructing the remodeling function in a video decoder according to one embodiment of this invention;
[028] Figures 6A and 6B show examples of how QP chroma shift values change according to the luma quantization parameter (QP) for signals encoded by PQ and HLG according to an embodiment of this invention; and [029] Figure 7 depicts an example of a pivot-based representation of a remodeling function according to an embodiment of this invention.
DESCRIPTION OF EXEMPLIFYING MODALITIES [030] This document describes techniques for remodeling and signal encoding integrated out of loop and loop. In the following description, for purposes of explanation, several specific details are presented in order to provide a full understanding of the present invention. However, it will be evident that the present invention can be practiced without these specific details. In other cases, well-known structures and devices will not be described in
Petition 870200044274, of 07/04/2020, p. 12/88
6/76 in-depth details, in order to avoid unnecessarily occluding, obscuring or obscuring the present invention.
OVERVIEW [031] The exemplary modalities described in this document refer to the remodeling and signal encoding integrated for video. In an encoder, a processor receives an input image in a first codeword representation represented by an input bit depth Ne an input codeword mapping (for example, gamma, PQ, and the like). The processor selects an encoder architecture (with a remodeler being an integral part of the encoder) from two or more candidate encoder architectures to compress the input image using a second codeword representation allowing for more efficient compression than first code word representation, where the two or more candidate encoder architectures comprise an out-of-loop remodeling architecture, an intra-loop only remodeling architecture, or a loop architecture for prediction residuals, and the compact processor the input image according to the selected encoder architecture.
[032] In another embodiment, a decoder to generate output images in the first codeword representation receives a coded bit stream with at least part of the coded images being compressed into the second codeword representation. It also receives associated remodeling information. The processor receives a signal indicating the decoder architecture from two or more candidate decoder architectures to decompress the encoded bitstream input, where the two or more candidate decoder architectures comprise an out-of-loop remodeling architecture, a remodeling architecture only intra looped images, or a
Petition 870200044274, of 07/04/2020, p. 13/88
7/76 loop architecture for prediction residuals, and unzips the encoded image to generate an output image according to the remodeling architecture received.
[033] In another modality, in an encoder to compress images according to a loop architecture for prediction residuals, the processor accesses an input image in a first code word representation and generates a progressive reshaping function that maps pixels of the input image from the first code word representation to the second code word representation. It generates an inverse remodeling function based on the progressive remodeling function that maps pixels from the second code word representation to pixels in the first code word representation. Then, for an input pixel region in the input image: it computes at least one predicted region based on pixel data in a reference frame buffer or previously encoded spatial neighbors;
generates a remodeled residual region based on the input pixel region, the predicted region and the progressive remodeling function;
generates a coded residual region (transformed and quantized) based on the remodeled residual region;
generates a decoded residual region (inverse and transformed quantized) based on the encoded residual region;
generates a reconstructed pixel region based on the decoded residual region, the predicted region, the progressive remodeling function, and the inverse remodeling function; and generates a reference pixel region to be stored in the reference frame buffer based on the reconstructed pixel region.
[034] In another modality, in a decoder to generate output images in the first code word representation according to an architecture
Petition 870200044274, of 07/04/2020, p. 14/88
8/76 in loop for prediction residuals, the processor receives an encoded bit stream partially encoded in the second codeword representation. It also receives associated remodeling information. Based on the remodeling information, the processor generates a progressive remodeling function that maps pixels from the first codeword representation to the second codeword representation and an inverse remodeling function, in which the inverse remodeling function maps pixels to from the second code word representation to the first code word representation. For a region of the encoded image, the processor:
generates a remodeled residual region decoded based on the encoded image;
generates a predicted region based on pixels in a reference pixel buffer or previously decoded spatial neighbors;
generates a reconstructed pixel region based on the decoded remodeled residual region, the predicted region, the progressive remodeling function and the inverse remodeling function;
generates an output pixel region based on the reconstructed pixel region; and, stores the output pixel region in the reference pixel buffer.
EXAMPLE VIDEO IMPLANTATION PROCESSING PIPELINE [035] Figure 1A depicts an exemplary process of a conventional video deployment pipeline (100) that shows various stages from video capture to the display of video content. A sequence of video frames (102) is captured or generated using an image block (105). Video frames (102) can be digitally captured (for example, by a digital camera) or generated by a computer (for example, using computer animation) for
Petition 870200044274, of 07/04/2020, p. 15/88
9/76 providing video data (107). Alternatively, the video frames (102) can be captured on film by a film camera. The film is converted into a digital format to provide video data (107). In a production phase (110), the video data (107) is edited to provide a video production flow (112).
[036] The production stream video data (112) is then provided to a processor in the block (115) for post-production editing. The post-production diction block (115) may include adjusting or modifying colors or brightness in particular areas of an image to enhance image quality or achieve a particular appearance for the image in accordance with the creative purpose of the video creator. It is sometimes referred to as "color timing" or "color grading." Other edits (for example, selection and scenes and continuity, cropping images, adding computer-generated visual special effects, etc.) can be performed in block (115) to produce a final version (117) of production for distribution. During post-production editing (115), video images are displayed on a reference screen (125).
[037] Following post-production (115), final production video data (117) can be deployed to the encoding block (120) to deploy downstream to decoding and reproducing devices, such as television sets, decoders signs, cinemas, and the like. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other implantation formats, to generate an encoded bit stream (122). At a receiver, the encoded bit stream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) representing an identical or close approximation of the signal (117). The receiver can be attached to a target screen (140) which may have completely different characteristics from the reference screen (125). In this case, a block of
Petition 870200044274, of 07/04/2020, p. 16/88
10/76 display management (135) can be used to map the dynamic range of the decoded signal (132) to the characteristics of the target screen (140) generating a mapped display signal (137).
Signal reshaping [038] Figure 1B depicts an exemplary process for signal reshaping according to the prior art Ref. [1], Given the input frames (117), a progressive reshaping block (150) analyzes the input and the encoding restrictions and generates code word mapping functions that map input frames (117) to requanted output frames (152). For example, the input (117) can be encoded according to a certain electro-optical transfer function (EOTF) (for example, gamma). In some embodiments, information about the remodeling process can be communicated to downstream devices (such as decoders) using metadata. Depending on the usage in question, the term “metadata” refers to any auxiliary information that is transmitted as part of the encoded bit stream and assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, spatial color or gamma information, reference screen parameters, and auxiliary signal parameters, as described in this document.
[039] After encoding (120) and decoding (130), decoded frames (132) can be processed by a retrograde (or inverse) reshaping function (160), which converts the requanted frames (132) back into one original EOTF domain (for example, gamma), for further downstream processing, such as the display management process (135) discussed earlier. In some embodiments, the retrograde reshaping function (160) can be integrated with a decoder in the decoder (130), for example, as part of the decoder in an AVC or HEVC video decoder.
[040] Depending on the usage in question, the term “remodeler” can denote a
Petition 870200044274, of 07/04/2020, p. 17/88
11/76 progressive or inverse reshaping function to be used when encoding and / or decoding digital images. Examples of remodeling functions are discussed in Ref. [1] and [2]. For the purposes of this invention, it is assumed that an individual skilled in the art can derive progressive and inverse remodeling functions appropriately according to the characteristics of the input video and the available bit depth of the encoding and decoding architectures.
[041] In Ref. [1], a method of image reshaping based on block-in-loop for high-dynamic range video encoding has been proposed. This design allows for a block-based remodeling within the coding loop, but at the cost of increased complexity. More specifically, the design requires maintaining two sets of decoded image buffers: one set for decoded figurations inversely remodeled (or not remodeled), which can be used for prediction without remodeling and for output to a screen, and another set for figurations decoded progressively remodeled, which is used only for prediction without remodeling. Although decoded figures progressively remodeled can be computed quickly, the cost of complexity is very high, especially for interpretation (motion compensation with subpixel interpolation). In general, display image buffer (DPB) management is complicated and requires very careful attention, so, as assessed by the inventors, simplified methods for encoding video are desired.
[042] The modalities of remodeling-based codec architectures presented in this document can be divided as follows: an architecture with a remodeler outside the external loop, an architecture with a remodeler only intra-loop, and an architecture with a remodeler in loop for prediction residuals, also referred to, in abbreviation. As ‘residual loop remodeler’. A video encoder or decoder can support any of these architectures or a combination of them. Each of these architectures
Petition 870200044274, of 07/04/2020, p. 18/88
12/76 can also be applied in its own combination with any of the others. Each architecture can be applied to the luminance component, a chroma component, or a combination of the luma components and one or more chroma components.
[043] In addition to these three architectures, additional modalities describe efficient signaling methods for remodeling-related metadata, and various encoder-based optimization tools to improve coding efficiency when a remodel is applied.
Out-of-loop normative remodeler [044] Figures 2A and 2B depict architectures for a video encoder (200A_E) and a corresponding video decoder (200A_D) with a “normative” out-of-loop remodeler. The term "normative" denotes that, unlike previous designs where a remodeling was considered a preprocessing step, therefore, outside the normative description of a coding standard, such as AVC, HEVC, and the like, in this modality, the progressive remodeling and reverse are part of the normative requirements. Unlike the architecture in Figure 1B, where a bitstream compliance according to a standard is tested after decoding (130), in Figure 2B, compliance is tested after the reverse remodeling block (265) (for example, in exit 162 in Figure 1B).
[045] In the encoder (200A_E), two new blocks are added to a traditional block-based encoder (for example, HEVC): a block (205) to estimate the progressive remodeling function, and the progressive figuration remodeling block ( 210), which applies progressive remodeling to one or more color compounds of the input video (117). In some modalities, these two operations can be performed as part of a single image remodeling block. The parameters (207) related to the determination of the reverse remodeling function in the decoder can be passed to the encoder block without loss of the
Petition 870200044274, of 07/04/2020, p. 19/88
13/76 video encoder (for example, CABAC 220) so that they can be incorporated into the encoded bit stream (122). All operations related to intra or interpredition (225), transform and quantization (T & Q), inverse transform and quantization (Q ^-1 & T ^-1 ) and loop filtering, are performed using remodeled figures stored in DPB (215) .
[046] In the decoder (200A_D), two new normative blocks are added to a traditional block-based decoder: a block (250) to reconstruct a reverse remodel function based on coded remodel function parameters (207), and a block (265) to apply the inverse reshaping function to the decoded data (262) to generate the decoded video signal (162). In some embodiments, operations related to blocks 250 and 265 can be combined into a single processing block.
[047] Figure 3A depicts an exemplary process (300A_E) for encoding video using the out-of-loop remodeling architecture (200A_E) according to an embodiment of this invention. If remodeling is not enabled (path 305), then the coding proceeds as known and prior art coders (for example, HEVC). If a remodel is enabled (trajectory 310), then an encoder may have the options to apply a given (default) remodel function (315), or adaptively determine a new remodel function (325) based on an analysis (320) (for example, as described in references [1] - [3]). After the progressive remodeling (330), the rest of the coding follows the traditional coding pipeline (335). If an adaptive remodel (312) is employed, metadata related to the reverse remodel function is generated as part of the “Coding Remodel” step (327).
[048] Figure 3B depicts an example process (300A_D) for decoding video using the out-of-loop remodeling architecture (200A_D) of
Petition 870200044274, of 07/04/2020, p. 20/88
14/76 according to an embodiment of this invention. If no remodeling is enabled (trajectory 355), then, after decoding a picture (350), output frames are generated (390) as in a traditional decoding pipeline. If a remodel is enabled (trajectory 360), then, in step (370), the decoder determines whether to apply a predetermined remodeling function (default) (375), or adaptively determines the inverse remodeling function (380) based on parameters received (for example, 207). After reverse remodeling (385), the rest of the decoding follows the traditional decoding pipeline.
Remodeling only below in normative loop [049] Figure 2C depicts an exemplary architecture for an encoder (200B_E) using normative loop remodeling only in accordance with one embodiment of this invention. The design is very similar to the design proposed in Ref. [1]; however, to reduce complexity, especially since it refers to the use of DPB memory (215 and 260), only pictures are coded using this architecture.
[050] Comparing to out-of-loop remodeling (200A_E), the main difference in the 200B_E encoder is that DPB (215) stores reverse remodeled figures instead of remodeled figures. In other words, the remodeled intra-remodeling figures need to be remodeled inversely (by the 265 reverse remodeling unit) before being stored in the DPB. The reason behind this approach is that if intra figures are coded with remodeling, the improved performance of intra figure coding will propagate to improve (implicitly) the coding of inter figures too, only that two inter figures are coded without remodeling. Thus, one can take advantage of remodeling without dealing with the complexity of loop remodeling for inter figurations. Since an inverse remodeling (265) is part of the inner loop, it can be implemented before the loop filter (270). The advantage of adding
Petition 870200044274, of 07/04/2020, p. 21/88
15/76 reverse remodeling before the loop filter is that in this case the loop filter design can be optimized based on the characteristics of the original figures instead of the progressive remodeled figures.
[051] Figure 2D depicts an exemplary architecture for a decoder (200B_D) using normative loop remodeling only intra according to one embodiment of this invention. As shown in Figure 2D, determining the reverse remodeling function (250) and applying reverse remodeling (265) will now be carried out before loop filtering (270).
[052] Figure 3C depicts an exemplary process (300B_E) for encoding video using a loop-only remodeling architecture according to one embodiment of this invention. As pictured, the flow of operations in Figure 3C shares many elements with the flow of operations in Figure 3A. Now, by default, remodeling for intercoding is not applied. For intracoded figures, if a remodel is enabled, an encoder again has the option of using a standard remodeling curve or applying adaptive remodeling (312). If a figuration is remodeled, an inverse remodeling (385) is part of the process and associated parameters are coded in step (327). The corresponding decoding process (300B_D) is shown in Figure 3D.
[053] As shown in Figure 3D, operations related to remodeling are enabled only for received intra figures and only if an intra remodeling is applied to the encoder.
Loop remodeling for prediction residuals [054] In coding, the term 'residual' denotes the difference between a prediction of a sample or data element and its original or decoded value. For example, given an original sample of the input video (117), denoted as Orig_sample, intraprediction or interpredition (225) it can generate a corresponding predicted sample (227) denoted as Pred_sample. If there is no
Petition 870200044274, of 07/04/2020, p. 22/88
16/76 remodeling, the non-remodeled residual (Res_u) can be defined as
Res_u = Orig_sample - Pred_sample. (1) [055] In some embodiments, remodeling in the residual domain may be beneficial. Figure 2E depicts an exemplary architecture for an encoder (200C_E) using loop remodeling for predicted residuals according to an embodiment of this invention. Let's say that Fwd () denotes the progressive remodeling function and let's say that InvQ denotes the corresponding inverse remodeling function. In one embodiment, a refurbished residual (232) can be defined as
Res_r = Fwd (Orig_sample) - Fwd (Pred_sample). (2) [056] Correspondingly, at the exit (267) of the inverse remodeler (265), the reconstructed sample, denoted as Reco_sample (267), can be expressed as
Reco_sample = lnv (Res_d + Fwd (Pred_sample)), (3) where Res_d represents the residual (234), an approximation close to Res_r, after loop encoding and decoding at 200C_E.
[057] Note that although reshaping is applied to residuals, the actual input video pixels are not reshaped. Figure 2F shows the corresponding decoder (200C_D). Note that as shown in Figure 2F, and based on equation (3), a decoder requires access to the progressive and inverse remodeling functions, which can be extracted using received metadata (207) and the “Remodeling Decoder” block ( 250).
[058] In one mode, to reduce complexity, equations (2) and (3) can be simplified. For example, assuming that the progressive remodeling function can be approximated by a linear function in sections and that the absolute difference between Pred_sample and Orig_sample is relatively small, then equation (2) can be approximated as
Res_r = a (Pred_sample) / Orig_sample -Pred_sample), (4)
Petition 870200044274, of 07/04/2020, p. 23/88
17/76 where (Pred_sample) denotes a scaling factor based on the value of Pred_sample. From equations (3) and (4), equation (3) can be approximated as
Reco_sample = Pred_sample + (' / a (Pred_sample)) * Res_r, (5)
Therefore, in a modality, it is necessary to communicate with a decoder only the scaling factors a (Pred_sample) for the linear model in sections.
[059] Figures 3E and 3F depict exemplary process flows for encoding (300C_E) and decoding (300C_D) a video using loop reshaping of prediction residuals. The processes are very similar to those described in Figures 3A and 3B, and therefore self-explanatory.
[060] Table 1 summarizes the main features of the three proposed architectures.
Table 1: Key features for remodeling architectures under consideration
^ '' '' ^ grqu itetu ra Out of loop Loop Intra Only Loop residual DPB storage Remodeled figures Intra mode: reverse remodeled figures Inter mode: no remodeling Unmodified figures Intra prediction performed Remodeled figures Remodeled figures Unmodified figures Inter prediction (movement estimate) performed Remodeled figures Unmodified figures Unmodified figures Additional figuration buffer required yes (buffer is required to keep figures remodeled in DPB and figures not remodeled for output) no (quick replacement of figuration samples) no (quick replacement of residual samples)
Petition 870200044274, of 07/04/2020, p. 24/88
18/76
Adaptive remodeling estimate location / frequency Unrestricted (can be only intra, scene based or configurable Only in intra figurations Unrestricted (can be intra-only, scene-based, or configurable) Complexity for sample modification (remodeling) Processes all figurations Processes only intra figurations (less complex) Processes residuals, either intra or inter Loop filter interaction Optimization using a redesigned figuration as a reference Optimization using an original figuration as a reference Optimization using an original figuration as a reference Adaptive block / region level remodeling possible not not Yes Other aspects Inter prediction performance can be impaired if frames of reference have different remodeling functionsInter prediction can use a remodeler for current figuration to process residuals against reference frames (which may have different remodelers)The decoder side needs only a reverse remodeling function The decoder side needs only a reverse remodeling function The decoder needs progressive and reverse remodeling functions
[061] Figures 4A and 4B depict exemplary encoding and decoding processing flows for encoding and decoding using a combination of three proposed architectures. As depicted in Figure 4A, if a remodel is not enabled, the incoming video is encoded according to known video encoding techniques (for example, HEVC and the like) without using any remodeling. Otherwise, the encoder can select any of the three main proposed methods, depending on the capabilities of the target receiver and / or the input characteristics. For example, in one embodiment, an encoder can switch between these methods at the scene level, where a 'scene' is denoted as a sequence of continuous frames with characteristics of
Petition 870200044274, of 07/04/2020, p. 25/88
19/76 similar luminance. In another modality, high-level parameters are defined at the Sequence Parameter Set (SPS) level.
[062] As depicted in Figure 4B, a decoder, depending on the signaling received from the remodeling information, can invoke any of the corresponding decoding processes to decode the input encoded bit stream.
Loop loop remodeling [063] Figure 2G depicts an exemplary architecture (200D_E) for an encoder using a loop loop remodeling architecture. This architecture combines elements of the remodeling architectures only intra loop (200B_E) and residual loop (200C_E) discussed previously. Under this architecture, the Intra slices are coded according to the intra loop remodeling coding architecture (for example, 200B_E in Figure 2C), except for one differentiation: for Intra slices, a reverse figuration remodeling is performed (265 -1) after loop filtering (270-1). In another mode, loop filtering for Intra slices can be performed after an inverse remodeling; however, experimental results have shown that this arrangement can produce a worse coding efficiency than when an inverse remodeling is performed after loop filtering. The remaining operations remain the same as previously discussed.
[064] Inter slices are coded according to the residual loop coding architecture (for example, 200C_E in Figure 2E), as discussed earlier. As shown in Figure 2G, an Intra / Internter slice switch allows for switching between two architectures depending on the type of slice to be coded.
[065] Figure 2H depicts an exemplary architecture (200D_D) for a decoder using hybrid loop remodeling. Again, the Intra slices are decoded according to a remodeling decoder architecture
Petition 870200044274, of 07/04/2020, p. 26/88
20/76 intra loop (for example, 200B_D in Figure 2D), where, again, for intra slices, loop filtering (270-1) precedes the reverse figuration remodeling (265-1). Inter slices are decoded according to the residual loop decoding architecture (for example, 200C_D in Figure 2F). As shown in Figure 2H, an Intra / Internter slice switching allows for switching between the two architectures depending on the types of slice in the encoded video figures.
[066] Figure 4A can be easily extended to also include the hybrid loop remodeling coding method by invoking the 300D-E coding process depicted in Figure 2G. Similarly, Figure 4B can be easily extended to also include the hybrid loop remodeling decoding method by invoking the 300D-D decoding process depicted in Figure 2H.
Slice-level remodeling [067] The embodiments of the present invention allow for a variety of slice-level adaptations. For example, to reduce computations, remodeling can be enabled only for intra slices or only for inter slices. In another modality, remodeling can be allowed based on the value of a temporal ID (for example, HEVC Temporalld variable (Ref. [11]), where Temporalld = nuh_temporal_id_plus1 - 1). For example, if Temporalld for the current slice is less than or equal to a predefined value, then slice_reshaper_enable_flag for the current slice can be set to 1, otherwise slice_reshaper_enable_flag will be 0. To avoid sending the slice_reshaper_enable_flag parameter for each slice, if you specify the sps_reshaper_temporal_id parameter at the SPS level, then its value can be inferred.
[068] For slices where remodeling is enabled, the decoder needs to know which remodeling model to use. In one modality, one can always use the remodeling model defined at the SPS level. In another
Petition 870200044274, of 07/04/2020, p. 27/88
21/76 modality, you can always use the remodeling model defined in the slice header. If no remodeling model is defined on the current slice, then the remodeling model used on the most recently decoded slice that used remodeling can be applied. In another modality, the remodeling model can always be specified in Intra slices, regardless of whether remodeling is used for an intra slice or not. In this implementation, the slice_reshaper_enable_flag and slice_reshaper_model_present_flag parameters need to be decoupled. An example of this slice syntax is shown in Table 5.
Remodeling information signaling [069] Information related to progressive and / or reverse remodeling can be present in different information layers, for example, in the video parameter set (VPS), sequence parameter set (SPS), set of picture parameters (PPS), a slice header, supplementary information (SEI), or any other high-level syntax. As an example, and without limitation, Table 2 provides an example of a high level syntax in the SPS for signaling whether remodeling is enabled, whether remodeling is adaptive or not, and which of the three architectures is being used.
Table 2: Example of remodeling information in SPS
SPS () Descriptor ......sps_reshaper_enable_flag TV. remodeling on, if not off * / u (1) if (sps_reshaper_enable_flag) {sps_reshaper_adaptive_flag / * 1: adaptive remodeling is on, if not 7 u (1) sps_reshaper_architecture / * for example ,: 0: out of loop, 1: intra in loop, 2: residual in loop 7 eu (v) }......
[070] Additional information can also be transmitted in some other layer, say in the slice header. The remodeling functions can be described by lookup tables (LUT), polynomials in stretches, or other types
Petition 870200044274, of 07/04/2020, p. 28/88
22/76 of parametric models. The type of remodeling model being used to communicate the remodeling functions can be signaled by additional syntax elements, for example, a reshaping_model_type flag. For example, we consider a system that uses two different representations: model_A (for example, reshaping_model_type = 0) represents the remodeling function as a set of polynomials in segments (for example, see Ref. [4]), while in model_B (for example, reshaping_model_type = 1) the remodeling function is derived adaptively by assigning code words to different luminance bands based on the figuration luminance characteristics and visual importance (for example, see Ref. [3]). Table 3 provides an example of syntax elements in the slice header of a picture to assist a decoder in determining the appropriate remodeling model being used.
Table 3: Exemplary syntax for remodeling signaling in a slice header
slice_segment_header () Descriptor ......if (sps_reshaper_adaptive_flag) {reshaping_model_type eu (v) if (reshaping model type == model A) {reshaping_sliceheader_table_model_A ()}otherwise if (reshaping_model_type == model_B) {reshaping_sliceheader_table_model_B ()}if no...}......
[071] The following three Tables describe alternative examples of a bit stream syntax for signal reshaping in the Sequence, Slice, or Coding Tree Unit (CTU) layers.
Table 4: Example of remodeling information in SPS
SPS () Descriptor ......sps_reshaper_enable_flag / * 1: remodeling on, if not u (1)
Petition 870200044274, of 07/04/2020, p. 29/88
23/76
off * /if (sps reshaper enable flag) {sps_reshaper_signal_type / * 0: SDR, 1: PQ, 2: HLG * / u (2) sps_reshaper_ILF_opt / * loop filter in which domain: 2 bits inter / intra * / u (2) sps_reshaper_chromaAdj / * 1: chromaDQP; 2: chroma stagger / u (2) sps_reshaper_model_present_flag / * 1: present * / u (D if (sps_reshaper_model_present_flag)sps reshaper model ()}
Table 5: Exemplary syntax for remodeling signaling in a slice header___________________________________________________________
slice_header () Descriptor ......slice_reshaper_model_present_flag u (D se (slice reshaper model present flag)slice reshaper model ()slice_reshaper_enable_flag u (1) if (slice reshaper enable flag) {reshaper_CTU_control_flag / * 1: on, CTU level on / off flag * 1 u (1) }
Table 6: Exemplary syntax for remodeling signaling in a
CTU
coding tree unit () Descriptor ......if (reshape CTU control flag) {reshaper CTU flag ae (v) }
[072] For Tables 4 to 6, the example semantics can be denoted as:
sps_reshaper_enable_flag equal to 1 specifies that a remodeler is used in the encoded video sequence (CVS). sps_reshaper_enabled_flag equal to 0 specifies that a remodeler is not used in CVS.
Petition 870200044274, of 07/04/2020, p. 30/88
24/76 slice_reshaper_enable_flag equal to 1 specifies that a remodeler is enabled for the current slice. slice_reshaper_enable_flag equal to 0 specifies that a remodeler is not enabled for the current slice.
sps_reshaper_signal_type indicates the distribution or representation of original code words. As an example, and without limitation, sps_reshaper_signal_type equal to 0 specifies SDR (gamma); sps_reshaper_signal_type equal to 1 specifies PQ; and sps_reshaper_signal_type equal to 2 specifies HLG.
reshaper_CTU_control_flag equal to 1 indicates that a remodeler is allowed to be adapted for each CTU. reshaper_CTU_control_flag equal to 0 indicates that a remodeler is not allowed to be adapted for each CTU. When reshaper_CUT_control_flag is not present, the value must be inferred to be 0.
reshaper_CTU_flag equal to 1 specifies that a remodeler is left for the current CTU. reshaper_CUT_flag equal to 0 specifies that a remodeler is not used for the current CTU. When reshaper_CTU_flag is not present, the value must be inferred equal to slice_reshaper_enabled_flag.
sps_reshaper_model_present_flag equal to 1 indicates that sps_reshaper_model () is present in sps. sps_reshaper_model_present_flag equal to 0 indicates that sps_reshaper_model () is not present in SPS.
slice_reshaper_model_present_flag equal to 1 indicates that slice_reshaper_model () is present in the slice header. slice_reshaper_model_present_flag equal to 0 indicates that slice_reshaper_model () is not present in SPS.
sps_reshaper_chromaAdj equal to 1 indicates that a chroma QP adjustment is performed using chromaDQP. sps_reshaper_chromaAdj equal to 2 indicates that the chroma QP adjustment is performed using chroma scaling.
Petition 870200044274, of 07/04/2020, p. 31/88
25/76 sps_reshaper_ILF_opt indicates whether the loop filter should be applied in the original domain or in the remodeled domain for intra and inter slices. For example, use a two-bit syntax, where the least significant bit refers to intra slices:
sps_reshaper_l LF_opt Loop filter operations 00 In the original domain for both intra and inter 0 1 In the original domain for inter, in a remodeled domain for intra 1 0 In remodeled domain for inter, in original domain for intra 1 1 In a remodeled domain for both intra and inter
[073] In some modalities, this parameter can be adjusted at the slice level. For example, in one mode, a slice can include a slice_reshape_ILFOPT_flag when slice_reshaper_enable_flag is set to 1. In another mode, in SPS, one can include a sps_reshaper_ILF_Tid parameter if sps_reshaper_ILF_opt is enabled. If TemporallD for current slice <= sps_reshaper_ILF_Tid and slice_reshaper_enable_flag is set to 1, then the Loop Filter is applied in the remodeling domain. Otherwise, it is applied to the unmodified domain.
[074] In Table 4, the chroma QP adjustment is controlled at the SPS level. In one embodiment, the QP chroma adjustment can also be controlled at the slice level. For example, in each slice, you can add the slice_reshape_chromaAdj_flag syntax element when slice_reshaper_enable_flag is set to 1. In another mode, in SPS, you can add the sps_reshaper_ChromaAdj_Tid syntax element if sps_reshaper_chromaAdj is enabled. If TemporallD for current slice <= sps_reshaper_ChromaAdj_Tid and slice_reshaper_enable_flag is set to 1, then a chroma adjustment is applied. Otherwise, the chroma adjustment is not applied. Table 4B depicts an exemplary variation from Table 4 using the syntax described above.
Table 4B: Exemplary syntax for remodeling signaling in
Petition 870200044274, of 07/04/2020, p. 32/88
26/76
SPS using temporal IDs
SPS () Descriptor ......sps_reshaper_enable_flag / * 1: remodeling on, if not off * / u (1) if (sps_reshaper_enable_flag) {sps_reshaper_signal_type / * 0: HDR, 1: PQ, 2: HLG * / u (2) sps_reshaper_ILF_opt / * loop filter in which domain: 2 bits inter / intra * / u (2) if (sps_reshaper_ILF_opt == 3)sps_reshaper_ILF_Tid u (3) sps_reshaper_chromaAdj / * 1: chromaDQP; 2: chromaScaling / u (2) if (sps_reshaper_chromaAdj)sps_reshaper_chromaAdj_Tid u (3) sps_reshaper_model_present_flag / * 1: present * / u (D se (sps reshaper model present flag)sps_reshaper_model ()}
sps_reshaper_ILF_Tid specifies the largestTemporallD where a loop filter is applied to a remodeled slice in the remodeled domain.
sps_reshaper_chromaAdj_Tid specifies the largest TemporallD to which a chroma adjustment is applied for a remodeled slice.
[075] In another modality, the remodeling model can be defined using a remodeling model ID, for example, reshape_model_id, for example, as part of the slice_reshape_model () function. The remodeling model can be signaled at SPS, PPS or slice header levels. If flagged in SPS or PPS, the value of reshape_model_id can also be inferred from sps_seq_parameter_set_id or pps_pic_parameter_set_id. An example of how to use reshape_model_id for slices other than slice_reshape_model () (for example, slice_reshaper_model_present_flag equal to 0) is shown below in Table 5B, a variation of Table 5.
Table 5B: Example syntax for remodeling signaling in a slice header using reshape_model_id
Petition 870200044274, of 07/04/2020, p. 33/88
27/76
slice_header () Descriptor ......slice_reshaper_model_present_flag u (1) if (slice_reshaper_model_present_flag)slice reshaper model ()if noreshape_model_id eu (v) slice_reshaper_enable_flag u (1) if (slice_reshaper_enable_flag) {reshaper_CTU_control_flag / * 1: on, CTU level on / off flag * / u (1) }......
In example syntax, the reshape_model_id parameter specifies the value for the reshape_model being used. The reshape_model_id value must be in the range 0 to 15.
[076] As an example of the proposed syntax, an HDR signal encoded using the PQ EOTF is considered, where remodeling is used at the SPS level, no specific remodeling is used at the slice level (remodeling is used for all slices), and CTU adaptation is allowed only for Inter slices. So:
sps_reshaper_signal_type = 1 (PQ);
sps_reshaper_model_present_flag = 1;
// Note: You can manipulate the slice_reshaper_enable_flag to enable and disable the remodeler for inter slices.
slice_reshaper_enable_flag = 1;
if (CTUAdp) {
if (l_slice) slice_reshaper_model_present_flag = 0;
reshaper_CTU_control_flag = 0;
if no
Petition 870200044274, of 07/04/2020, p. 34/88
28/76 slice_reshaper_model_present_flag = 0;
reshaper_CTU_control_flag = 1;
} if no {
slice_reshaper_model_present_flag = 0;
reshaper_CTU_control_flag = 0;
} [077] In another example, an SDR level is considered where remodeling is applied only at the slice level, and only for Intra slices. A CTU remodeling adaptation is only allowed for Inter slices. So:
sps_reshaper_signal_type = 0 (SDR);
sps_reshaper_model_present_flag = 0;
slice_reshaper_enable_flag = 1;
if (l_slice) {
slice_reshaper_model_present_flag = 1;
reshaper_CTU_control_flag = 0;
} if no {
slice_reshaper_model_present_flag = 0;
if (CTUAdp) reshape_CTU_control_flag = 1;
otherwise reshaper_CTU_control_flag = 0;
Petition 870200044274, of 07/04/2020, p. 35/88
29/76}
[078] At the CTU level, in one modality, a CTU level remodeling can be enabled based on the luminance characteristics of the CTU. For example, for each CTU, one can compute the average luminance (for example, CTU_avg_lum_value), compare it with one or more thresholds, and decide to turn the remodeling on or off based on the results of these comparisons. For example, if CTU_avg_lum_value <THR1, or if CTU_avg_lum_value> THR2, or if THR3 <CTU_avg_lum_value <THR4, then reshaper_CTU_Flag = 1 for that CTU.
In one modality, instead of using the average luminance, you can use some other luminance characteristic of CTU, such as the minimum, maximum or average luminance, variance, and the like. CTU characteristics can also be applied, or it can be combined with luminance and chroma characteristics and thresholds.
[079] As previously described (for example, in relation to the steps in Figures 3A, 3B and 3C), the modalities can support a standard or static remodeling function, or adaptive remodeling. A “standard remodeler” can be used to perform a predefined remodeling function, therefore, reducing the complexity to analyze each figuration or scene in deriving a remodeling curve. In this case, there is no need to signal an inverse remodeling function at the scene, figuration, or slice level. The standard remodeler can be implemented using a fixed mapping curve stored in the decoder to avoid any signaling, or it can be signaled once as part of the sequence level parametric adjustments. In another modality, an adaptive remodeling function previously decoded can be reused for later figurations in order of coding. In another modality, the remodeling curves can be signaled differently in relation to those
Petition 870200044274, of 07/04/2020, p. 36/88
30/76 previously decoded. In other modalities, (for example, for residual loop remodeling where the / nv () and Fwd () functions are necessary to perform an inverse remodeling), only one of the / nv () functions can be signaled in the bit stream or FwdÇ), or alternatively, to reduce decoder complexity, both. Tables 7 and 8 provide two examples for signaling remodeling information.
[080] In Table 7, the remodeling function is communicated as a set of second order polynomials. It is a simplified syntax of the Exploratory Test Model (ETM) (Ref. [5]). An earlier variation can also be found in Ref. [4],
Table 7: Exemplary syntax for representing parts of a remodeling function (model_A)
reshaping_sliceheader_table_model_A () { Described r reshape input luma bit depth minus8 eu (v) coeff_log2_offset_minus2 eu (v) reshape_num_ranges_minus1 eu (v) reshape equal ranges flag u (1) reshape_global_offset_val u (v) if (! reshape_equal_ranges_flag)for (i = 0; i <reshape_num_ranges_minus1 + 1; i ++)reshape range val [i] u (v) reshape continuity flag u (1) for (i = 0; i <reshape in a minusl ranges + 2; i ++) {reshape_poly_coeff_orderO_int [i] eu (v) reshape poly coeff orderO fracfi] u (v) }if (reshape continuity flag = = 1) {reshape poly coeff orderl int if (v) reshape poly coeff orderl frac u (v) }____________________________________________1____________________________________________
reshape_input_luma_bit_depth_minus8 specifies the sample bit depth of the input luma component of the remodeling process.
coeff_log2_offset_minus2 specifies the number of fractional bits for calculations of remodeling-related coefficients for the luma component. O
Petition 870200044274, of 07/04/2020, p. 37/88
31/76 coeff_log2_offset_minus2 value must be in the range 0 to 3, inclusive.
reshape_num_ranges_minus1 plus 1 specifies the number of tracks in the stretch remodeling function. When not present, the value of reshape_num_ranges_minus1 is inferred to be 0. reshape_num_ranges _minus1 must be in the range 0 to 7, including for the luma component.
reshape_equal_ranges_flag equal to 1 specifies that the remodeling function in sections is partitioned into sections of NumberRanges with an almost equal length and the length of each strip is not explicitly flagged. reshape_equal_ranges_flag equal to 0 specifies that the length of each band is explicitly flagged.
reshape_global_offset_val is used to derive the offset value that is used to specify the starting point of the 0 th range.
reshape_range_val [i] is used to derive the length of the i-th range of the luma component.
reshape_continuity_flag specifies the continuity properties of the remodel function for the luma component. If reshape_continuity_flag is equal to 0, a zero-order continuity is applied to the linear inverse remodeling functions in stretches between consecutive pivot points. If reshape_continuity_flag is equal to 1, first order smoothing is used to derive the complete second order polynomial inverse remodeling functions between consecutive pivot points.
reshape_poly_coeff_orderO_int [i] specifies the integer value of the polynomial coefficient of 0-th order of i-th in parts for luma component.
reshape_poly_coeff_orderO_frac [i] specifies the fractional value of the polynomial coefficient of 0-th order of i-th in a stretch for the luma component.
reshape_poly_coeff_order1_int specifies the entire value of the coefficient
Petition 870200044274, of 07/04/2020, p. 38/88
32/76 first order polynomial for luma component.
reshape_poly_coeff_order1_frac specifies the fractional value of the first order polynomial coefficient for luma component.
[081] Table 8 describes an exemplary modality of an alternative parametric representation according to model_B discussed previously (Ref. [3]).
Table 8: Exemplary syntax for parametric representation of a remodeling function (model_B)
reshaping sliceheader table model B () { Described r reshape model profile type eu (v) reshape model scale idx u (2) reshape_model_min_bin_idx u (5) max bin idx reshape model u (5) reshape_model_num_band u (4) for (i = 0; i <reshape model in a band; i ++) {reshape_model_band_profile_delta [i] u (1) }____________________________________________} ____________________________________________
[082] In Table 8, in a modality, syntax parameters can be defined as:
reshape_model_profile_type specifies the type of profile to be used in the remodeling construction process.
reshape_model_scale_idx specifies the index value of a scalar factor (denoted as ScaleFactor) to be used in the remodeling construction process. The ScaleFactor value allows for improved control of the reshaping function for improved overall coding efficiency. Additional details on the use of this ScaleFactor are provided in relation to the discussion of the remodeling function reconstruction process (for example, as described in Figures 5A and 5B). As an example, and without limitation, the reshape_model_scale_idx value should be in the range 0 to 3, inclusive. In one embodiment, the mapping relationship between scalejdx and ScaleFactor as shown in the Table below is given by
Petition 870200044274, of 07/04/2020, p. 39/88
33/76
ScaleFactor = 1.0 - 0.05 * reshape mode scalejdx.
reshape_model_scale_idx ScaleFactor 0 1.0 1 0.95 2 0.9 3 0.85
In another modality, for a more efficient fixed point implementation,
ScaleFactor = 1 - 1/16 * reshape_model_scale_idx.
reshape_model_scale_idx ScaleFactor 0 1.0 1 0.9375 2 0.875 3 0.8125
reshape_model_min_bin_idx specifies the minimum binary index to be used in the remodeling construction process. The reshape_model_min_bin_idx value must be in the range 0 to 31, inclusive.
reshape_model_max_bin_idx specifies the maximum binary index to be used in the remodeling construction process. The reshape_model_max_bin_idx value must be in the range 0 to 31, inclusive.
reshape_model_num_band specifies the number of bands to be used in the remodeling construction process. The reshape_model_num_band value must be in the range 0 to 15, inclusive.
reshape_model_band_profile_delta [i] specifies the delta value to be used to adjust the i-band profile in the remodeling construction process. The reshape_model_band_profile_delta [i] value must be in the range 0 to 1, inclusive.
[083] Compared to Ref. [3], the syntax in Table 8 is much more efficient by defining a set of “standard profile types”, say, highlights, intermediate and dark tones. In one mode, each type has a pre-defined visual band importance profile. The predefined bands and corresponding profiles can be implemented as fixed values in the decoder or can also be
Petition 870200044274, of 07/04/2020, p. 40/88
34/76 signaled using a high level syntax (such as parametric string set). In the encoder, each image is first analyzed and categorized into one of the profile types. The profile type is signaled by the syntax element “reshape_model_profile_type.” In adaptive remodeling, in order to capture the full range of image dynamics, the standard profiling is additionally adjusted by a delta for each or a subset of luminance bands. Delta values are derived based on the visual importance of the luminance bands, and are signaled by the syntax elements “reshape_model_band_profile_delta.” [084] In a modality, the delta value can only assume the values of 0 or 1. In the encoder, the visual importance is determined by comparing the percentage of band pixels in the entire image with the percentage of band pixels within “Dominant bands,” where dominant bands can be detected using a local histogram. If pixels within a band are concentrated in a small local block, the band is most likely to be important visual in the block. The counts for dominant bands are added and normalized to form a meaningful comparison to obtain delta values for each band.
[085] In a decoder, a remodeling function reconstruction process needs to be invoked to derive the remodeling LUTs based on methods described in Ref. [3], Therefore, the complexity is higher compared to the model of approximation in sections simple, which only needs to evaluate the polynomial functions in stretches to compute the LUT. The benefit of using a parametric model syntax is that it can significantly reduce the bit rate of using a remodeler. For example, based on typical test content, the model depicted in Table 7 needs 200 to 300 bits to signal a remodeler, while a parametric model (as shown in Table 8) uses only about 40 bits.
Petition 870200044274, of 07/04/2020, p. 41/88
35/76 [086] In another modality, as shown in Table 9, the progressive remodeling research table can be derived according to a parametric model for the dQP values. For example, in a modality, dQP = clip3 (m / n, max, scale * X + offset), where min and max denote the limits of dQP, scale and offset are two parameters of the model, and X denotes a derived parameter based on signal luminance (for example, a pixel luminance value, or for blocks, a block luminance metric, for example, its minimum, maximum, mean, variance, standard deviation, and the like). For example, without limitation, dQP = clip3 (-3, 6, 0.015 * X-7.5).
Table 9: Example syntax for parametric representation of a remodeling function (model C)
sps_reshaper_model_C () { descriptor full_range_input_flag u (1) dQP_model_scale_int_prec eu (v) if (dQP_model_scale_int_prec> 0) {dQP_model_scale_int u (v) }dQP_model_scale_frac_prec_minus16 eu (v) dQP_model_scale_frac u (v) if (dQPModeIScaleAbs) {dQP_model_scale_sign u (1) }dQP_model_offset_int_prec_minus3 eu (v) dQP_model_offset_int u (v) dQP_model_offset_frac_prec_minus1 eu (v) dQP_model_offset_frac u (v) if (dQPModelOffsetAbs) {dQP_model_offset_sign u (1) }dQP_model_abs_prec_minus3 eu (v) dQP_model_max_abs u (v) if (dQP_model_max_abs) {dQP_model_max_sign u (1) }dQP_model_min_abs u (v)
Petition 870200044274, of 07/04/2020, p. 42/88
36/76
if (dQP_model_min_abs) {dQP_model_min_sign u (1) }}
[087] In one embodiment, the parameters in Table 9 can be defined as follows:
full_range_input_flag specifies the input video signal range. A full_range_input_flag of 0 corresponds to a standard dynamic range input video signal. A full_range_input_flag of 1 corresponds to the full-range input video signal. When not present, full_range_input_flag is inferred to be 0.
Note: Depending on the usage in question, the term “full-range video” denotes that valid code words in the video are not “limited.” For example, for a 10-bit full-range video, valid code words are between 0 and 1023, where 0 is mapped to the lowest luminance level. In contrast, for 10-bit “standard range video”, valid code words are between 64 and 940, and 64 is mapped to the lowest luminance level.
For example, the calculation of “full range” and “standard range” can be computed as follows:
for normalized luminance values Ey 'in [0 1], encode in BD bits (for example, BD = 10, 12, and the like):
full range: Y = clip3 (0, (1 «BD) - 1, Ey '* ((1« BD) -1))) standard range: Y = clip3 (0, (1 «BD) - 1, round ( 1 «(BD-8) * (219 * Ey '+ 16)))
This syntax is similar to the “video_full_range_flag” syntax in HEVC VUI parameters as described in Section E.2.1 of the HEVC Specification (H.265) (Ref. [11]) · dQP_model_scale_int_prec specifies the number of bits used for the representation of dQP_model_scale_int. A dQP_model_scale_int_prec equal to 0 indicates that dQP_model_scale_int is not flagged and is inferred to be 0.
Petition 870200044274, of 07/04/2020, p. 43/88
37/76 dQP_model_scale_int specifies the dQP model scale integer value.
dQP_model_scale_frac_prec_minus16 plus 16 specifies the number of bits used for the representation of dQP_model_scale_frac.
dQP_model_scale_frac specifies the fractional value of the dQP model scale
The dQPModeIScaleAbs variable is derived as:
dQPModeIScaleAbs = dQP_model_scale_int «(dQP_model_scale_frac_prec_minus16 + 16) + dQP_model_scale_frac dQP_model_scale_sign specifies the dQP model scale signal. When dQPModeIScaleAbs is equal to 0, dQP_model_scale_sign is not flagged and is inferred to be 0.
dQP_model_offset_int_prec_minus3 plus 3 specifies the number of bits used for the representation of dQP_model_offset_int.
dQP_model_offset_int specifies the entire dQP model offset value.
dQP_model_offset _frac_prec_minus1 plus 1 specifies the number of bits used for the representation of dQP_model_ offset_frac.
dQP_model_offset_frac specifies the fractional value of the dQP model offset.
The dQPModelOffsetAbs variable is derived as:
dQPModelOffsetAbs = dQP_model_offset_int «(dQP_model_offset_frac_prec_minus1 + 1) + dQP_model_offset_frac dQP_model_offset_sign specifies the dQP model offset signal. When dQPModelOffsetAbs is equal to 0, dQP_model_offset_sign is not flagged and is inferred to be 0.
dQP_model_abs_prec_minus3 plus 3 specifies the number of bits used for the representation of dQP_model_max_abs and dQP_model_min_abs.
Petition 870200044274, of 07/04/2020, p. 44/88
38/76 dQP_model_max_abs specifies the dQP model maximum integer value. dQP_model_max_sign specifies the dQP model maximum signal. When dQP_model_max_abs is equal to 0, dQP_model_max_sign is not flagged and is inferred to be 0.
dQP_model_min_abs specifies the dQP model minimum integer value.
dQP_model_min_sign specifies the dQP model minimum sign. When dQP_model_min_abs is equal to 0, dQP_model_min_sign is not flagged and is inferred to be 0.
Decoding process for Model C [088] Given the syntax elements in Table 9, the remodeling LUT can be derived as follows.
The dQPModeIScaleFP variable is derived as:
dQPModeIScaleFP = ((1- 1 * dQP_model_scale_sign) * dQPModeIScaleAbs) «(dQP_model_offset_frac_prec_minus1 + 1).
The dQPModelOffsetFP variable is derived as:
dQPModelOffsetFP = (((1-2 * dQP_model_offset_sign) * dQPModelOffsetAbs) «(dQP_model_scale_frac_prec_minus16 + 16).
The dQPModelShift variable is derived as:
dQPModelShift = (dQP_model_offset_frac_prec_minus1 + 1) + (dQP_model_scale_frac_prec_minus16 + 16).
The dQPModelMaxFP variable is derived as:
dQPModelMaxFP = ((1- 2 * dQP_model_max_sign) * dQP_model_max_abs) «dQPModelShift.
The dQPModelMinFP variable is derived as:
dQPModelMinFP = ((1- 2 * dQP_model_min_sign) * dQP_model_min_abs) «dQPModelShift.
for Y = 0: maxY // For example, for 10-bit video, maxY = 1023
Petition 870200044274, of 07/04/2020, p. 45/88
39/76 {
dQP [Y] = clip3 (dQPModelMinFP, dQPModelMaxFP, dQPModelScaleFP * Y + dQPModelOffsetFP);
slope [Y] = exp2 ((dQP [Y] +3) / 6); // fixed point exp2 implementation where exp2 (x) = 2 ^Λ (χ);
} if (fuil_range_input_flag == 0) // if the input is a standard range video For Y out of the standard range (ie, Y = [0:63] and [940: 1023]), define slope [Y] = 0;
CDF [0] = slope [0];
for Y = 0: maxY-1 {
CDF [Y + 1] = CDF [Y] + slope [Y]; // CDF [Y] is the slope integral [Y]}
for Y = 0: maxY {
FwdLUT [Y] = round (CDF [Y] * maxY / CDF [maxY]); // rounding and normalization to obtain FwdLUT}
[089] In another modality, as shown in Table 10, the progressive remodeling function can be represented as a collection of luma pivot points (ln_Y) and their corresponding code words (Out_Y). To simplify coding, the input luminance range is described in terms of a starting pivot and a sequence of subsequent pivots equally spaced using a linear representation in sections. An example of a representation of a progressive remodeling function for 10-bit input data is shown in Figure 7.
Table 10: Exemplary syntax for pivot-based representation of
Petition 870200044274, of 07/04/2020, p. 46/88
40/76 a remodeling function (model D)
sps_reshaper_model_D () { descriptor full_range_input_flag u (1) bin_pivot_start u (v) bin_cw_start u (v) Iog2_num_equal_bins_minus3 eu (v) equal_bin_pivot_delta u (v) bin_cw_in_first_equal_bin u (v) bi n_cw_de lta_a bs_prec_mi n us4 eu (v) for (i = 0; i <NumEqualBins -1; i ++) {bin_cw_delta_abs [i] u (v) if (bin_cw_delta_abs [i]) {bin_cw_delta_sign [i] u (D }}}
[090] In one embodiment, the parameters in Table 10 can be defined as follows:
full_range_input_flag specifies the input video signal range. A full_range_input_flag of 0 corresponds to the standard range input video signal. A full_range_input_flag of 1 corresponds to the full-range input video signal. When it is not present, full_range_input_flag is inferred to be 0.
bin_pivot_start specifies the pivot value of the first equal length binary (710). When full_range_input_flag is equal to 0, bin_pivot_start must be greater than or equal to the smallest standard range entry, and must be less than the largest standard range entry. (For example, for a 10-bit SDR entry, bin_pivot_start (710) must be between 64 and 940).
bin_cw_start specifies the mapped value (715) of bin_pivot_start (710) (for example, bin_cw_start = FwdLUT [bin_pivot_start]).
Iog2_num_equal_bins_minus3 plus 3 specifies the number of torques of equal length subsequent to the starting pivot (710). The variables NumEqualBins and NumTotalBins are defined by:
Petition 870200044274, of 07/04/2020, p. 47/88
41/76
NumEqualBins = 1 «(log2_num_equal_bins_minus3 + 3) if f ull_range_in put_flag == 0
NumTotalBins = NumEqualBins + 4 otherwise
NumTotalBins = NumEqualBins + 2
Note: Experimental results show that the progressive remodeling functions can be represented using eight segments of equal length; however, complex remodeling functions may require more segments (for example, 16 or more).
equal_bin_pivot_delta specifies the length of binaries of equal length (for example, 720-1, 720-N). NumEqualBins * equal_bin_pivot_delta must be less than or equal to a valid input range. (For example, if full_range_input_flag is 0, a valid input range should be 940-64 = 876 for 10-bit inputs; if full_range_input_flag is 1, a valid input range should be 0 to 1023 for 10-bit inputs.) bin_cw_in_first_equal_bin specifies the number of mapped code words (725) in the first binary of equal length (720-1).
bin_cw_delta_abs_prec_minus4 plus 4 specifies the number of bits used for the representation of bin_cw_delta_abs [i] for each subsequent equal binary.
bin_cw_delta_abs [i] specifies the value of bin_cw_delta_abs [i] for each subsequent binary of equal length. bin_cw_delta [i] (for example, 735) is the difference in code words (for example, 740) in the current equal length binary i (for example, 720-N) compared to the code words (for example, 730) in previous equal-length torque i-1.
bin_cw_delta_sign [i] specifies the bin_cw_delta_abs [i] signal. When bin_cw_delta_abs [i] is equal to 0, bin_cw_delta_sign [i] is not flagged and is inferred to be 0.
Petition 870200044274, of 07/04/2020, p. 48/88
42/76
The variable bin_cw_delta [i] = (1-2 * bin_cw_delta_sign [i]) * bin_cw_delta_abs [i].
Decoding Process for Model D [091] Given the syntax element in Table 10, the remodeling LUT can be derived as follows for a 10-bit entry:
Definite constants:
minIN = minOUT = 0;
maxIN = maxOUT = 2 ^A BD - 1 = 1023 for 10 bits // BD = Bit depth minStdIN = 64 for 10 bits maxStdIN = 940 for 10 bits
Step 1: Derive a pivot value ln_Y [j] forj = 0: NumTotalBins ln_Y [0] = 0;
ln_Y [NumTotalBins] = maxIN;
if (full_range_input_flag == 0) {
ln_Y [1] = minStdIN;
ln_Y [2] = bin_pivot_start;
for (j = 3: NumT ota I Bi ns - 2) ln_Y [j] = ln_Y [j -1] + equal_bin_pivot_delta;
ln_Y [NumTotalBins - 1] = maxStdIN;
} if no {
ln_Y [1] = bin_pivot_start;
for j = 2: NumTotalBins -1 ln_Y [j] = ln_Y [j -1] + equal_bin_pivot_delta;
}
Step 2: Derive a mapped value Out_Y [j] forj = 0: NumTotalBins
Petition 870200044274, of 07/04/2020, p. 49/88
43/76
Out_Y [Ο] = 0;
Out_Y [NumTotalBins] = maxOUT;
if (full_range_input_flag == 0) {
Out_Y [1] = 0;
Out_Y [2] = bin_cw_start;
Out_Y [3] = bin_cw_start + bin_cw_in_first_equal_bin;
bin_cw [3] = bi n_cw_in_first_equal_bin;
for j = (4: NumT ota I Bi ns - 2) bin_cw [j] = bin_cw [j -1] + bin_cw_delta [j - 4]; // bin_cw_delta [i] start from idxO to j = (4: NumT ota I Bi ns - 2)
Out_Y [j] = Out_Y [j -1] + bin_cw [j];
Out_Y [NumTotalBins - 1] = maxOUT;
} if no {
Out_Y [1] = bin_cw_start;
Out_Y [2] = bin_cw_start + bin_cw_in_first_equal_bin;
bin_cw [2] = bin_cw_in_first_equal_bin;
for j = (3: NumT ota I Bi ns - 1) bin_cw [j] = bin_cw [j -1] + bin_cw_delta [j - 3]; // bin_cw_delta [i] start from idxO to j = 3: NumTotalBins -1
Out_Y [j] = Out_Y [j -1] + bin_cw [j];
}
Step 3: linear interpolation to obtain all LUT entries
Petition 870200044274, of 07/04/2020, p. 50/88
44/76
Init: FwdLUT [] for (j = 0: NumTotalBins - 1) {
InS = ln_Y [j];
InE = ln_Y [j + 1];
OutS = Out_Y [j];
OutE = Out_Y [j + 1];
for i = ln_Y [j]: ln_Y [j + 1] -1 {
FwdLUT [i] = OutS + round ((OutE - OutS) * (i - InS) / (InE - InS));
}}
FwdLUT [ln_Y [NumTotalBins]] = Out_Y [NumTotalBins];
[092] In general, remodeling can be turned on or off for each slice. For example, you can enable remodeling for intra slices only and disable remodeling for inter slices. In another example, remodeling can be disabled for inter slices that have the highest time level. (Note: as an example, depending on the usage in question, the temporal sublayers can correspond to the definition of temporal sublayers in HEVC.) When defining the remodel model, in one example, you can flag only the remodel model in SPS, but in another example, we can signal the model of slice remodeler in intra slices. Alternatively, you can flag the remodeling model in SPS and allow the slice remodeling model to update the SPS remodeling model for all slices, or you can only allow the slice remodeling model to update the slicing model. SPS remodeler for intra slices. For inter slices that follow an intra slice, the SPS remodeler model or an intra slice remodeler model can be applied.
Petition 870200044274, of 07/04/2020, p. 51/88
45/76 [093] As another example, Figures 5A and 5B depict a process of reconstructing the remodeling function in a decoder according to a modality. The process uses the methods described in this document and in Ref. [3] with a visual rating range in [0 5], [094] As described in Figure 5A, first (step 510), the decoder extracts the variable reshape_model_profile_type and adjust (steps 515, 520 and 525) for each torque the appropriate starting band profile. For example, in pseudocode:
if (reshape_model_profile_type == 0) R [bi] = Rbright [bi];
otherwise if (reshape_model_profile_type == 1) R [bi] = Rdark [bi];
otherwise R [bi] = Rmid [bi].
[095] In step 530, the decoder adjusts each band profile using the received reshape_model_band_profile_delta [bi] values, as in (i = 0: reshape_model_num_band-1) {R [bi] = R [bi] + reshape_model_band_profile_delta [bi] }.
[096] In step 535, the decoder propagates the adjusted values for each binary profile, as if bin [j] will belong to bi band, R_bin [j] = R [bi], [097] In step 540, the binary profiles are modified, as if (j> reshape_model_max_bin_idx) or (j <reshape_model_min_bin_idx) {R_binU] = 0}.
[098] In parallel, in steps 545 and 550, the decoder can extract the parameters to compute the scale factor value and candidate code words for each bin [j], as in
ScaleFactor = 1.0- 0.05 * reshape_model_scale_idx
CW_dft [j] = code words in the binary if using a standard remodel CW_PQ [j] = TotalCW / TotalNumBins.
[099] In computing the ScaleFactor value, for a point implementation
Petition 870200044274, of 07/04/2020, p. 52/88
46/76 fixed, instead of using the 0.05 climber you can use 1/16 = 0.0625 in place.
[0100] Continuing with Figure 5B, in step 560, the decoder starts a code word (CW) pre-assignment for each binary based on the binary profile, as if R_bin [j] == 0, CW [ j] = 0 if R_bin [j] == 1, CW [j] = CW_dft [j] / 2;
if R_bin [j] == 2, CW [j] = min (CW_PQ [j], CW_dft [j]);
if R_bin [j] == 3, CW [j] = (CW_PQ [j] + CW_dft [j]) / 2;
if R_bin [j]> = 4, CW [j] = max (CW_PQ [j], CW_dft [j]);
[0101] In step 565, the total code words used are computed and the code word (CW) assignments are refined / completed, as in
CWused = Sum (CW [j]):
if CWused> TotalCW, rebalancing CW [j] = CW [j] / (CWused / TotalCW);
if no {
CW_remain = TotalCW - CWused;
CW_remain is assigned to binaries with the highest R_bin [j]);
} [0102] Finally, in step 565, decoder a) generates the progressive remodeling function (for example, FwdLUT) by accumulating the CW values [j], b) multiplies the ScaleFactor value by the FwdLUT values to form the final FwdLUT (FFwdLUT), and c) generates the InvLUT reverse remodeling function based on FFwdLUT.
[0103] In a fixed-point implementation, the computation of ScaleFactor and FFwdLUT can be expressed as:
ScaleFactor = (1 «SF_PREC) - reshape_model_scale_idx
FFwdLUT = (FwdLUT * ScaleFactor + (1 «(FP.PREC + SF.PREC -1)))»
Petition 870200044274, of 07/04/2020, p. 53/88
47/76 (FP_PREC + SF_PREC), where SF_PREC and FP_PREC are predefined variables related to precision (for example, SF_PREC = 4 and FP_PREC = 14), “c = a“ n ”denotes a binary left shift operation of a bit a by n (or c = a * (2 ⁿ )), and “c = a» rí 'denotes a binary right shift operation of a bit a by n (or c = a / (2 ⁿ )).
Chroma QP derivations [0104] Chroma coding performance is strictly related to luma coding performance. For example, in AVC and HEVC, a table is defined to specify the relationship between the quantization parameters (QP) for luma and chroma components, or between luminance and chrominance. The specifications also allow you to use one or more chroma QP offsets for additional flexibility in defining the QP relationship between luma and chroma. When a remodel is used, the luma value is modified, so the relationship between luminance and chrominance can also be modified. To maintain and further improve the coding efficiency under remodeling, in one mode, at the coding unit (CU) level, a chroma QP offset is derived based on the remodeling curve. This operation needs to be performed on both the decoder and the encoder.
[0105] Depending on the usage in question, the term “coding unit” (CU) denotes a coded block (for example, a macroblock and the like). For example, without limitation, in HEVC, a CU is defined as “a luma sample coding block, two corresponding chroma sample coding blocks of a picture that has three sample arrangements, or a sample coding block of a monochromic figuration or a figuration that is encoded using three separate color planes and syntax structures used to encode the samples. ”
Petition 870200044274, of 07/04/2020, p. 54/88
48/76 [0106] In one embodiment, the chroma quantization (QP) parameter value (chromaQP) can be derived as follows:
1) based on the remodeling curve, derive the dQP mapping from an equivalent luma, dQPLUT:
for CW = 0: MAX_CW_VALUE-1 dQPLUT [CW] = -6 * log2 (slope [CW]);
where slope [CW] denotes the slope of the progressive remodeling curve at each CW point (codeword), and MAX_CW_VALUE is the maximum codeword value for a given bit depth, for example, for a 10-bit signal, MAX_CW_VALUE = 1024 (2 ¹⁰ ).
Then, for each coding unit (CU):
2) compute the average luminance of the coding unit, denoted as AvgY:
3) compute the chromaDQP value based on dQPLUT [], AvgY, in the remodeling architecture, in the lnv () reverse remodeling function, and in the slice type, as shown in Table 11 below:
Table 11: Exemplary chromaDQP values according to remodeling architectures____________________________________________
Remodeling architecture Intra Slice Inter Slice Out of loop dQPLUT [lnv (AvgY)] dQPLUT [lnv (AvgY)] Loop-only remodeler dQPLUT [lnv (AvgY)] 0 Loop reshaper for residuals dQPLUT [AvgY] dQPLUT [AvgY] Loop hybrid remodeling dQPLUT [lnv (AvgY)] dQPLUT [AvgY]
4) compute chromaQP as:
chromaQP = QPJuma + chromaQPOffset + chromaDQP;
Petition 870200044274, of 07/04/2020, p. 55/88
49/76 where chromaQPOffset denotes a chroma QP offset, and QPJuma denotes the luma QP for the encoding unit. Note that the chroma QP offset value may be different for each chroma component (say, Cb and Cr) and the chroma QP offset values are communicated to the decoder as part of the encoded bit stream.
[0107] In one mode, dQPLUT [] can be implemented as a predefined LUT. It is assumed that all code words are divided into Λ / binary (for example, N = 32) and each binary contains M = MAX_CW_VALUE / N codewords (for example, M = 1024/32 = 32). When new code words are assigned to each binary, you can limit the number of code words to be 1 to 2 * M, so you can pre-compute dQPLUT [1 ... 2 * M] and save it as a LUT . This approach can avoid any floating point computations or the approximation of fixed point computations. You can also save encoding / decoding time. For each binary, a fixed chromaQPOffset is used for all code words in that binary. The DQP value is set to be equal to dQPLUT [L] where Léo is the number of code words for that binary, where 1 <L <2 * M.
The dQPLUT values can be pre-computed as follows:
for i = 1: 2 * M slopefi] = i / M;
dQPLUT [i] = -6 * log2 (slope [i]);
end
Different quantization schemes can be used to obtain an integer QP value when computing dQPLUTfx], such as: round (), ceil (), floor () or a mixture of them. For example, you can define a threshold TH, and if Y <TH, use floor () to quantize the value of dQP, otherwise, when Y> TH, use ceil () to quantize the value of dQP. The use of these quantization schemes and the corresponding parameters can be
Petition 870200044274, of 07/04/2020, p. 56/88
50/76 predefined in a codec or can be signaled in the bit stream for adaptation. An example syntax that allows a mixture of quantization schemes with a threshold as discussed above is shown as follows:
quant_scheme_signal_table () { Descriptor if (sps reshaper chromaAdj> 0) {quant_scheme_idc // 0: round (), 1: ceil (), 2: floor (), 3: mix u (2) if (quant_scheme_idc == 3) {// mixquant_change_threshold u (v) first_quant_scheme_idc u (2) second_quant_scheme_idc u (2) }}______________________________________________} ______________________________________________
The function quant_scheme_signal_table () can be defined at different levels of the remodeling syntax (for example, the sequence level, the slice level, and the like), depending on the need for adaptation.
[0108] In another mode, the chromaDQP values can be computed by applying a scaling factor to the residual signal in each coding unit (or transform unit, more specifically). This scaling factor can be a luma-dependent value and can be computed: a) numerically, for example, as the first order derivative (slope) of the progressive remodeling LUT (see, for example, equation (6) in Section below), or b) as:
AQP (x
Slope (x) = 6 <
When computing S / ope (x) using dQP (x), dQP can be maintained in floating point precision without integer quantization. Alternatively, one can compute quantized integer dQP values using a variety of different quantization schemes. In some modalities, this scaling can be performed at the pixel level instead of at the block level, where each chroma residue can be scaled by a different scaling factor, derived using the value of
Petition 870200044274, of 07/04/2020, p. 57/88
51/76 prediction of colocalized luma from this chroma sample. Soon,
Table 12: Exemplary chroma dQP values using scaling for remodeling architecture h _________________________
Update Intra Slice Inter Slice CU-based scheduling (same S_cu scheduler shared by all CU samples) Scu = SlopeLUT [lnv (AvgY)] C_Res_scaled = C_Res * S_cu Scu = SlopeLUT [AvgY] C_Res_scaled = C_Res * S_cu Pixel based chroma scaling (different S_px scaler on each sample) Spx = SlopeLUT [lnv (ColPredY)] C_Res_scaled = C_Res * S_px Spx = SlopeLUT [ColPredY] C_Res_scaled = C_Res * S_px
For example, if CSCALE_FP_PREC = 16 • Progressive scaling: after the chroma residual is generated, before transformation and quantization:
-C_Res = C_orig - C_pred
-C_Res_scaled = C_Res * S + (1 «(CSCALE_FP_PREC - 1)))» CSCALE_FP_PREC • Inverse scaling: after inverse quantization and inverse chroma transformation, but before reconstruction:
-C_Res_inv = (C_Res_scaled «CSCALE_FP_PREC) / S
-C_Reco = C_Pred + C_Res_inv;
where S is S_cu or S_px.
Note: In Table 12, in computing Yours, the average luma of a block (AvgY) is calculated before applying an inverse remodeling. Alternatively, an inverse remodeling can be applied before computing the average luma, for example,
Petition 870200044274, of 07/04/2020, p. 58/88
52/76
Scu = SlopeLUT [Avg (lnv [Y])]. This alternative order of computations applies to computation values in Table 11 as well; that is, computation of Inv (AvgY) can be replaced by computation of Avg values (lnv [Y]). The latter approach can be considered more accurate, but it has an increased computational complexity.
Encoder optimizations over remodeling [0109] This section discusses a number of techniques for improving encoding efficiency in the encoder by jointly optimizing the remodeling and encoder parameters when remodeling is a part of a normative decoding process (as described on one of the three candidate architectures). In general, coder optimization and redesign are facing the problem of coding in different locations with two limitations. In a traditional imaging and coding system, there are two types of quantization: a) sample quantization (for example, gamma coding or PQ) in the baseband signal and b) transform related quantization (part of the compression). The remodeling is located between them. The based remodeling and figuration is, in general, updated on a figuration basis and allows only sample value mappings based on their luminance level, without considering any spatial information. In a block-based codec (such as HEVC), transform quantization (for example, for luma) is applied to a spatial block and can be adjusted spatially, so encoder optimization methods need to apply the same set of parameters to a entire block containing samples with different luminance values. As assessed by the inventors and described in this document, joint remodeling and coding optimization can further improve coding efficiency.
Inter / lntra mode decision [0110] In traditional coding, inter / intra mode decisions are based on computing a distortion function (dft / nc ()) between the original samples and the
Petition 870200044274, of 07/04/2020, p. 59/88
53/76 predicted samples. Examples of these functions include the sum of quadratic errors (SSE), the sum of absolute differences (SAD), and others. In one embodiment, these distortion metrics can be used using redesigned pixel values. For example, if original dfunctÇ) uses Orig_sample (j) and Pred_sample (j), when remodeling is applied, dfunctÇ) can use their corresponding remodeled values, Fwd (Orig_sample (i)) and Fwd (Pred_sample (i)). This approach allows for a more precise inter / intra decision, thus improving coding efficiency.
LumaDQP with remodeling [0111] In the JCTVC HDR Common Test Conditions Document (CTC) (Ref. [6]), lumaDQP and chromaQPoffsets are two encoder settings used to modify quantization parameters (QP) for luma and chroma components to improved HDR encoding efficiency. In this invention, several new encoder algorithms are proposed to further refine the original proposal. For each lumaDQP fitting unit (for example, a 64x64 CTU), a dQP value is computed based on the unit's average input luma value (as in Table 3 in Ref. [6]). The final quantization parameter QP used for each Coding Unit within that lumaDQP adaptation unit must be adjusted by subtracting that dQP. The dQP mapping table is configurable in the encoder input configuration. This input configuration is denoted as dQPinp.
[0112] As discussed in Ref. [6] and [7], in existing coding schemes, the same lumaDQP LUT dQPinp is used for intra and inter figurations. Intra and inter figuration can have different properties and quality characteristics. In this invention, it is proposed to adapt the lumaDQP settings based on the type of figuration encoding. Therefore, two dQP mapping tables are configurable in the encoder input configuration, and are denoted as
Petition 870200044274, of 07/04/2020, p. 60/88
54/76 dQPinpIntra θ dQPinpInter [0113] As previously discussed, when using the Intra loop remodeling method, due to the fact that remodeling is not performed in inter figurations, it is important that some lumaDQP adjustment is applied to intercoded figures to achieve a similar quality as if the inter figurations were remodeled by the same remodeler used for intra figuration. In one embodiment, the lumaDQP setting for inter figurations must match the characteristics of the remodeling curve used in intra figurations. let's say
Slope (x) = Fwd '(x) = (Fwd (x + dx) - Fwd (x-dx)) / (2dx), (6) denotes the first derivative of the progressive remodeling function, then, in a modality, it is noted that the values of dQPauto (x) automatically derived can be computed as follows:
if S / ope (x) = 0, then (x) = 0, otherwise dQP _au to (x) = 6log2 (S / ope (x)), (7) where dQPauto (x) can be cut into a range reasonable, for example, [-6 6] · [0114] If lumaDQP is enabled for intra figurations with remodeling (ie, external dQPinpintra is adjusted), lumaDQP for inter figurations should be taken into account. In one modality, the interQual dQPtinai can be computed by adding the dQPauto derived from the remodeler (equation (7)) and the dQPinpintra adjustment for intra figures. In another modality, to take advantage of the propagation of intra quality, the dQPfinal for inter figurations can be adjusted as dQPauto or only in a small increment (adjusting dQPinpinter) and added to dQPauto · [0115] In one modality, when a remodeling enabled, the following general rules for adjusting lQ dQP values may apply:
(1) LQDQP mapping tables can be adjusted
Petition 870200044274, of 07/04/2020, p. 61/88
55/76 independently for intra and inter figurations (based on the type of figuration coding);
(2) If a figure within the coding loop is in the remodeled domain (for example, intra figures in Intra loop remodeling architecture or all figures in non-loop remodeling architecture), the input lumen for delta QP mapping , dQPinp, also needs to be translated into a refurbished dQP _rsp domain. That is DQP _rS p (x) = dQPin _P [LNV (x)]. (8) (3) If a figuration within the encoding loop is in an area not remodeled (e.g., refurbished inversely or not remodeled, e.g. , inter figurations in Intra loop remodeling architecture or all figurations in residual loop remodeling architecture), the input lumen to delta QP mapping does not need to be translated and can be used directly.
(4) Inter-automatic deltaQP derivation is only valid for Intra loop remodeling architecture. The actual delta QP used for inter figurations in this case consists of the summed values of autoderivative and input:
C / QPffna / [x] ⁼ dQPinp [x] + dQPauíofx], (9) and dQPfinai [x] can be trimmed to a reasonable range, for example [-12 12];
(5) The luma to the dQP mapping table can be updated in each figure or when there is a change in remodeling LUT. The adaptation of the real dQP (starting from an average luma value of the block, obtaining the corresponding dQP for quantizing that block) can happen at CU level (configurable encoder).
[0116] Table 13 summarizes the dQP adjustments for each of the three proposed architectures.
_______Table 13: dQP adjustment_______________________________________________
u itetu ra Out of loop Loop Intra Only Loop residual
Petition 870200044274, of 07/04/2020, p. 62/88
56/76
dQP of Intra figuration dQPfjnal (x) ⁼ dQPrsp (x) ⁼ dQPinplntra [lnv (x)] dQPfjnal (x) ⁼ dQPrsp (x) ⁼ dQPinplntra [lnv (x)] dQPfinal (x) - dQPinplntra (x) Inter figuration dQP dQPfinal (x) ⁼ dQPrsp (x) ⁼ dQPinplnter [l nv (x}] dQPfinal (x) ⁼ dQPauto (x) ⁺ dQPinplnte _________ x) _________ dQPfjnal (x) ⁼ dQPinplnter (x)
Rate distortion optimization (RDO) [0117] In the JEM6.0 software (Ref. [8]), a pixel based distortion of RDO (Rate Distortion Optimization) is used when lumaDQP is enabled. The weighting table is fixed based on luminance values. In one embodiment, the weighting table must be adaptively adjusted based on the lumaDQP adjustment, computed as proposed in the previous section. Two weights, for sum of quadratic error (SSE) and sum of absolute differences (SAD) are proposed as follows:
weight_SSE [x] = 2 ⁽ ^ ^-) , (10a) weight_SAD [x] = (10b) [0118] The weight computed by equation (10a) or equation (10b) is the total weight based on the final dQP, which comprises lumaDQP input and dQP derived from the progressive remodeling function. For example, based on equation (9), equation (10a) can be written as / / dQP i _n p [x] + dQP _aut0 [x] weig / it _SSF [x] = 2V ³ / = 2V ³ / / ^{d <} 2 ^p mpM zd <2P _aut0 [x] = 2v ³ M 3 λ
The total weight can be separated by weight computed by input lumaDQP: / dQP ínpM wej # / itss _E [x] -inp = 2v ³ /, and remodeling weight:
/ 6log2 (SZope (x)) / dQPgutoW 3 3 J „weight _SSE [x] _reshap e = 3) = 2 = (Slope (x) y.
When the total weight is computed using total dQP by computing the remodeling weight first, precision is lost by the clipping operation to obtain an entire dQPauto. Instead, directly using the tilt function to calculate the remodeling weight can preserve greater weight accuracy and,
Petition 870200044274, of 07/04/2020, p. 63/88
57/76 therefore, it is more favorable.
[0119] Denote as 1 / l / dQp the weight derived from the input lumaDQP. / '(%) Is said to denote the first derivative (or slope) of the progressive remodeling curve. In one embodiment, the total weight takes into account the dQP values and the shape of the remodeling curve, so a total weight value can be expressed as:
weight _tota i = Clip3 (0.0,30.0, W _dQP * / '(%) ² ). (11) [0120] A similar approach can also be applied to chroma components. For example, in a modality, for chroma, dQP [x] can be defined according to Table 13.
Interaction with other coding tools [0121] When remodeling is enabled, this section provides several examples of proposed changes needed in other coding tools. Interactions can exist for any possible existing or future encoding tools to be included in the next generation video encoding standard. The examples given below are not limiting. In general, the video signal domain (remodeled, not remodeled, inversely remodeled) during the coding steps needs to be identified and the operations that handle the video signal at each step need to take the remodeling effect into account.
Prediction of linear model of cross component [0122] In CCLM (prediction of linear model of cross component) (Ref. [8]), predicted chroma samples pred _c (i, j) can be derived using a light reconstruction signal rec _L '(i, j):
pred _c (i, f) = a rec _L '(i, f) + β. (12) [0123] When remodeling is enabled, in one mode, it may be necessary to distinguish whether the reconstructed luma signal is in the domain
Petition 870200044274, of 07/04/2020, p. 64/88
58/76 refurbished (for example, out-of-loop remodeler or intra-loop remodeler) or in non-remodeled domain (for example, residual loop remodeler). In one embodiment, the reconstruction luma signal can be implicitly used as it is without any additional signage or operation. In other modalities, if the reconstructed signal is in a non-remodeled domain, the reconstructed luma signal can be transferred to also be in the non-remodeled domain, as in:
pred _c (i, f) = a Inv (rec _L '(i,;)) + β. (13) [0124] In other modalities, elements of bitstream syntax can be added to signal which domain is desired (remodeled or not remodeled), which can be decided by an RDO process, or can be derived from decision based on the decoded information, thus saving the necessary overhead by explicit signaling. Operations corresponding to the reconstructed signal can be performed based on the decision.
Refurbisher with residual prediction tool [0125] The HEVC range extension profile includes a residual prediction tool. The residual chroma signal is predicted from the residual luma signal on the encoder side as:
(x, y) = r _c (x, y) - (ax / (x, y)) »3, (14) and is compensated on the decoder side as:
r ' _c (x, y) = kr' _c (x, y) + (axr / (x, y)) »3, (15) where r _c denotes the residual chroma sample in a position β denotes the residual sample reconstructed of the luma component, Ar _c denotes the predicted signal using inter-color precision, Ar / denotes the reconstructed signal after Δι · _Γ encoding and decoding, and r ' _c denotes the reconstructed chroma residual.
[0126] When remodeling is enabled, it may be necessary to consider which luma residual to use for residual chroma prediction. In a
Petition 870200044274, of 07/04/2020, p. 65/88
59/76 modality, you can use the “residual” as it is (it can be remodeled or not remodeled based on the remodeling architecture). In another modality, one can induce the luma residual to be in a domain (such as in the non-remodeled domain) and perform appropriate mappings. In another embodiment, proper handling can be derived by a decoder, it can be explicitly flagged as previously described.
Adaptive clipping remodeler [0127] Adaptive clipping (Ref. [8]) is a new tool introduced to signal a range of original data in relation to content dynamics, and to perform adaptive clipping instead of fixed clipping (based on information from internal bit depth) at each step in the compression workflow (for example, transform / quantize, loop filtering, output) where clipping takes place. let's say
Tciip ⁼ Clip _BD (T, bitdepth, C) = Clip3 (min _c , max _c , T), (16) where x = Clip3 (min, max, c) denotes: 'min, if c <min max, sec> max,, c, otherwise e
• C is the component ID (typically Y, Cb or Cr) • min _c is the lower clipping limit used on the current slice for component ID C • max _c is the upper clipping limit used on the current slice for component ID C [0128] When remodeling is enabled, in a modality, it may be necessary to verify the domain in which the data flow is currently located and perform the clipping correctly. For example, when dealing with data clipping and the refurbished domain, the original clipping limits need to be translated to the refurbished domain:
Petition 870200044274, of 07/04/2020, p. 66/88
60/76
T _cUp ⁼ Clip _BD (T, bitdepth, C) = = Clip3 (Fwd (min _c ), Fwd (maxc), 7). (17)
In general, you need to process each clipping step appropriately in relation to the remodeling architecture.
Loop Refurbishing and Filtering [0129] In HEVC and JEM software 6.0, loop filters, such as ALF and SAO, need to estimate the ideal filter parameters using reconstructed luma samples and unpacked “original” luma samples. When remodeling is enabled, in a modality, you can specify (explicitly or implicitly) the domain that you want to perform a filter optimization. In one modality, one can estimate the filter parameters in the remodeled domain (when the reconstruction is in the remodeled domain, versus a remodeled original). In other modalities, it is possible to estimate the filter parameters in the non-remodeled domain (when the reconstruction is in the non-remodeled or inversely remodeled domain, versus the original).
For example, depending on the loop remodeling architecture, the loop filter optimization (ILFOPT) options and operations can be described in Tables 14 and 15.
Table 14. Loop filtering optimization in loop-only remodeling architecture and hybrid loop remodeling
Intra Framework Inter Table Option 1: estimate and perform loop filtering (LF) in a remodeled domain; • Encoder side:o Use the reconstructed intra-original image as the LF reference o Use the reconstructed intra-image (in the remodeled domain) and the LF reference to estimate LF parameters; two cases, depending on the position of the LF module versus the • Encoder side:o Progressively reshape the original inter image as an LF referenceo Progressively reshape the inter-reconstructed image o Estimate LF parameters o Apply LF to the inter-reconstructed image in the remodeled domain
Petition 870200044274, of 07/04/2020, p. 67/88
61/76
reverse remodeling (blocks 265 and 270 in Figure 2C):if the reverse remodeling is performed before LF, you need to apply a progressive remodeling to the reconstructed intra imageif reverse remodeling needs to be performed after LF, use the reconstructed intra image directly o Apply LF to the reconstructed intra image in the remodeled domaino Conversely reshape the entire reconstructed intra imageo Store in DPB • Decoder side: o Apply LF to the reconstructed intra image in the remodeled domain; two cases, depending on the position of the LF module versus the reverse remodeling module (blocks 265 and 270 in Figure 2D): if the reverse remodeling is carried out before LF, you need to apply a progressive remodeling to the reconstructed intra image before applying LF if the reverse remodeling needs to be performed after LF, apply LF to the intra reconstructed image directlyo Conversely reshape the entire reconstructed intra imageo Store in DPB o Conversely reshape the entire inter-reconstructed image o Store in DPB • Decoder side: o Progressively reshape the inter-reconstructed image o Apply LF to the inter-reconstructed image in the remodeled domain o Conversely reshape the entire inter-reconstructed image o Store in DPB Option 2: estimate and perform LF in a non-domain • Encoder side: o Reshape the original image as (exact traditional LF workflow) • Encoder side:
Petition 870200044274, of 07/04/2020, p. 68/88
62/76 refurbished LF reference if a remodeling in position has been performed in the original image buffer; or search for an original, non-reshaped image as an LF reference o Reshape the reconstructed intra image o Estimate LF parameters o Inverse reshape the entire reconstructed image o Apply LF to the reconstructed intra image o Store in DPB • Decoder side: o Resolve the entire image the reconstructed image o Apply LF to the reconstructed intra image o Store in DPB o Use the inter-reconstructed image and the original image as LF reference to estimate LF parameters o Apply LF to the inter-reconstructed image o Store in DPB • Decoder side: o Apply LF to inter-reconstructed image o Store in DPB
Table 15. Loop filtering optimization in loop residual remodeling architecture
Residual loop remodeling Intra frame and Inter frame(same as the interframe operations in Table 14)
[0130] Although most of the discussions detailed in this document refer to methods performed on the luminance component, an individual skilled in the art will assess what similar methods can be performed on chroma color components and chroma-related parameters, such as chromaQPOffset (for example , see Ref. [9]).
Loop reshaping and regions of interest (ROD [0131] Given an image, depending on the usage in question, the term 'region of interest' (ROI) denotes a region of the image that is considered to be of special interest. innovative modalities that support loop remodeling for regions of interest only. That is, in one modality, remodeling can only be applied within an ROI and not outside. In another modality, different remodeling curves can be applied in one
Petition 870200044274, of 07/04/2020, p. 69/88
63/76 region of interest and outside the region of interest.
[0132] The use of ROls is motivated by the need to balance the bit rate and image quality. For example, a video sequence of a sunset is considered. In the upper half of the images, you can have the sun in the sky of relatively uniform color (therefore, pixels at the bottom of the sky can have a very low variance). In contrast, the bottom half of the image can depict moving waves. From an observer's perspective, from the top it can be considered much more important than the bottom. On the other hand, moving waves, due to the higher variance in their pixels, are more difficult to compress, requiring more bits per pixels; however, you may want to allocate more bits in the sun part than in the wave part. In this case, the upper half can be denoted as the region of interest.
ROI description [0133] Currently, most codecs (for example, AVC, HEVC, and the like) are based on blocks. In order to make the implementation simpler, you can specify the region in units of blocks. Without limitation, using HEVC as an example, a region can be defined as a multiple of Coding Units (CUs) or Coding Tree Units (CTUs). You can specify an ROI or multiple ROls. Multiple ROls can be distinct or overlapping. An ROI does not have to be rectangular. The syntax for ROls can be provided at any level of interest, such as the slice level, the figuration level, the video stream level, and the like. In one embodiment, the ROI is specified first in the sequence parameter set (SPS). So, in a slice header, small variations in ROI can be allowed. Table 16 depicts an example of a syntax where an ROI is specified as multiple CTUs in a rectangular region. Table 17 describes the syntax for a modified ROI at the slice level.
Table 16: SPS Syntax for ROI
SPS () { Described r
Petition 870200044274, of 07/04/2020, p. 70/88
64/76
sps reshaper enable flag u (D if (sps_reshaper_enable_flag) {.....sps_reshaper_active_ROI_flag u (D if (sps reshaper active ROI flag) {reshaper_active_ROI_in_CTUsize_left eu (v) reshaper_active_ROI_in_CTUsize_right eu (v) reshaper_active_ROI_in_CTUsize_top eu (v) reshaper active ROI in CTUsize bottom eu (v) }} }
Table 17: ROI slice header syntax
reshaping sliceheader tableQ { Described r if (sps reshaper active ROI flag) {reshape model ROI modification flag u (D if (reshape model ROI modification flag) {reshaper ROI mod offset left if (v) reshaper_ROI_mod_offset_right if (v) ROI reshaper mod offset top if (v) reshaper_ROI_mod_offset_bottom if (v) }} ____________________________________________} ____________________________________________
sps_reshaper_active_ROI_flag equal to 1 specifies that ROI exists in the encoded video sequence (CVS). sps_reshaper_active_ROI_flag equal to 0 specifies that ROI does not exist in CVS.
reshaper_active_ROI_in_CTUsize_left, reshaper_active_ROI_in_CTUsize_right, reshaper_active_ROI_in_CTUsize_top and reshaper_active_ROI_in_CTUsize_bottom specify the sample of the figures in
ROI, in terms of a rectangular region specified in figurative coordinates.
The coordinates are equal to an offset * CTUsize for left and top and offset * CTUsize-1 for right and bottom.
Petition 870200044274, of 07/04/2020, p. 71/88
65/76 reshape_model_ROI_modification_flag equal to 1 specifies that ROI is modified in the current slice. reshape_model_ROI_modification_flag equal to 0 specifies that ROI is not modified on the current slice.
reshaper_ROI_mod_offset_left, reshaper_ROI_mod_offset_right, reshaper_ROI_mod_offset_top and reshaper_ROI_mod_offset_bottom specify the left / right / top / bottom displacement value from reshaper_active_ROI_in_CT_ize_in_ize_in_ize_resize_ resize_ resize
For multiple ROls, the example syntax for Tables 16 and 17 for a single ROI can be extended using an index (or ID) for each ROI, similar to the scheme used in HEVC to define using multiple pan-scan rectangles of SEI messages (see the HEVC specification, Ref. [11], Section D.2.4).
Processing of ROls in remodeling only Intra loop [0134] For remodeling only intra, the ROI part of the figuration is remodeled first, then coding is applied. Due to the fact that the remodeling is applied only to ROI, a limit can be observed between the parts of ROI and non-ROI of a figuration. Since a loop filter (for example, 270 in Figure 2C or Figure 2D) may exceed the limits, special care must be taken for ROls for loop filter optimization (ILFOPT). In one embodiment, it is proposed that the loop filter be applied only where all the decoded figuration is in the same domain. In other words, the complete picture is all in the remodeled domain or all in the non-remodeled domain. In one embodiment, on the decoder side, if loop filtering is applied to an unmodified domain, an inverse remodeling must first be applied to the ROI section of the decoded figuration, and then apply the loop filter. The decoded figuration is then stored in the DPB. In another modality, if the loop filter is applied to the remodeled domain, a remodeling must be applied to the non-ROI part of the figuration first.
Petition 870200044274, of 07/04/2020, p. 72/88
66/76 decoded, then apply the loop filter, and then inversely reshape the entire figuration. The decoded figuration is then stored in the DPB. In yet another modality, if loop filtering is applied to the remodeled domain, you can inversely remodel the ROI part of the decoded figuration, then remodel the entire figuration, then apply the loop filter, then inversely remodel the entire figuration. The decoded figuration is then stored in the DPB. The three approaches are summarized in Table 18. From a computing point of view, method “A” is the simplest. In one embodiment, ROI enablement can be used to specify the order to perform reverse remodeling versus loop filtering (LF). For example, if ROI is actively used (for example, SPS syntax flag = true), then LF (block 270 in Figure 2C and Figure 2D) is performed after a reverse remodeling (block 265 in Figure 2C and Figure 2D). If ROI is not actively used, then LF is performed before reverse remodeling.
Table 18. Loop filtering (LF) options using ROls
Method A Method B Method C • Inversely reshape the ROI of a decoded figure • Apply loop filtering (LF) to the entire figure • Store in DPB • Apply remodeling to the non-ROI part of a decoded figure • Apply LF to the entire figure • Reverse remodel the entire figure • Store in DPB • Inversely reshape the ROI of a decoded picture • Reshape the entire picture • Apply LF to the entire picture • Reshape the entire picture inversely • Store in DPB
Processing of ROls in loop order residual remodeling [0135] For a loop residual remodeling (prediction) architecture (for example, see 200C_D in Figure 2F), in the decoder, using equation (3), the processing can be expressed as:
Petition 870200044274, of 07/04/2020, p. 73/88
67/76 if (currentCTU belongs to ROI)
Reco_sample = lnv (Res_d + Fwd (Pred_sample)), (see equation (3)) otherwise
Reco_sample = Res_d + Pred_sample order
ROls and Encoder Considerations [0136] In an encoder, each CTU needs to be checked whether it belongs to an ROI or not. For example, for residual loop prediction remodeling, a simple check based on equation (3) can perform:
if (currentCTU belongs to ROI) apply RDO-weighted distortion to luma. Weight is derived based on equation (10) if not applying an unweighted RDO distortion to an end [0137] An exemplary coding workflow that takes ROls into account during remodeling can comprise the following steps:
-For an Intra figure:
-Apply progressive remodeling to the ROI area of the original figuration
-Code the Intra frame
-Apply reverse remodeling to the ROI area of the reconstructed figure before the loop filter (LF)
-Realize loop filtering in the non-modeled domain as follows (for example, see method “C” in Table 18), which includes the following steps:
• Apply progressive remodeling to the non-ROI area of the original figuration (in order to make the entire original figuration remodeled to the loop filter reference) • Apply progressive remodeling to the entire figuration area of the figuration
Petition 870200044274, of 07/04/2020, p. 74/88
68/76 reconstructed • Derive the loop filter parameters and apply loop filtering • Apply reverse remodeling to the entire figuration area of the reconstructed figuration and store in DPB
On the encoder side, since LF needs to have an uncompressed reference image for filter parameter estimation, the reference treatment of
LF for each method is as shown in Table 19:
Table 19. LF reference treatment for ROls
Method A Method B Method C • Use figuration of original entrance not remodeled for reference of LF; • Apply remodeling throughout (non-ROI and ROI part) of the original entry figure for LF reference; • Apply remodeling throughout (non-ROI and ROI part) ad figuration of original entry for LF reference;
-For an Inter figuration:
-When coding an inter frame, for each CU within ad ROI, apply residual remodeling of prediction and weighted distortion and luma; for each CU outside the ROI, do not apply any remodeling
- Loop filter optimization (option 1) is performed as before (as if no ROI had been used):
• Progressively reshape the entire figuration area of the original figuration • Progressively reshape the entire figuration area of the reconstructed figuration • Derive the loop filter parameters and apply loop filtering • Apply reverse remodeling to the reconstructed figuration area and store in DPB
Reshaping of the content encoded by HLG [0138] The term HybridLog-Gamma or HLG denotes another transfer function defined in Rec. BT. 2100 to map high dynamic range signals.
Petition 870200044274, of 07/04/2020, p. 75/88
69/76
HLG was developed to maintain backward compatibility with traditional standard dynamic range signals encoded using the traditional gamma function. When comparing the codeword distribution between PQ-encoded content and HLG-encoded content, PQ mapping tends to allocate more codewords in dark and bright areas, while most HLG content codewords appear to be allocated in the middle range. Two approaches can be used for remodeling an HLG luma. In one embodiment, you can simply convert the HLG content into PQ content and then apply all of the PQ-related remodeling techniques discussed earlier. For example, the following steps can be applied:
1) Map HLG luminance (for example, Y) to PQ luminance. Let's say the function or LUT of the transformation is denoted as HLG2PQLUT (Y)
2) Analyze the luminance values of PQ and derive a progressive remodeling function based on PQ or LUT. Denote it as PQAdpFLUT (Y)
3) Join the two functions or LUTs into a single function or LUT: HLGAdpFLUT [i] = PQAdpFLUT [HLG2PQLUT [i]].
[0139] Since the HLG code word distribution is very different from the PQ code word distribution, this approach can produce sub-ideal remodeling results. In another modality, the HLG remodeling function is derived directly from the HLG samples. You can apply the same framework as used for PQ signals, but change the CW_Bins_Dft table to reflect the characteristics of an HLG signal. In one embodiment, using the intermediate tone profile for HLG signals, several CW_Bins_Dft Tables are designed according to the user's preferences. For example, when it is preferred to preserve highlights, for alpha = 1.4, g_DftHLGCWBinO = {8, 14, 17, 19,21,23, 24, 26, 27, 28,29, 30, 31,32, 33,
34, 35, 36, 36, 37, 38, 39, 39, 40, 41, 41, 42, 43, 43, 44, 44, 30}.
Petition 870200044274, of 07/04/2020, p. 76/88
70/76
When it is preferable to preserve the intermediate tones (or intermediate range): g_DftHLGCWBin1 = {12, 16, 16, 20, 24, 28, 32, 32, 32, 32, 36, 36, 40, 44, 48, 52, 56, 52, 48, 44, 40, 36, 36, 32, 32, 32, 26, 26, 20, 16, 16, 12}.
When it is preferable to preserve skin tones:
g_DftHLGCWBin2 = {12, 16, 16, 24, 28, 32, 56, 64, 64, 64, 64, 56, 48, 40, 32, 32, 32, 32, 32, 32, 28, 28, 24, 24 , 20, 20, 20, 20, 20, 16, 16, 12};
[0140] From a syntax point of view, to differentiate between remodeling based and PQ and HLG, a new parameter, denoted as sps_reshaper_signal_type, is added, where the value sps_reshaper_signal_type indicates the type of signal that has been remodeled (for example, 0 for gamma-based SDR signals, 1 for PQ encoded signals, and 2 for HLG encoded signals).
[0141] Examples of syntax tables for remodeling HDR in SPS and slice header for PQ and HLG, with all the features discussed previously (for example, ROI, in loop filter optimization (ILFOPT), and ChromaDQPAdjustment), are shown in Tables 20 and 21.
Table 20: Exemplary SPS syntax for remodeling
SPS () Descriptor ......sps_reshaper_enable_flag / * 1: remodeling on, if not off * / u (1) if (sps_reshaper_enable_flag) {sps_reshaper_adaptive_flag / * 1: adaptive remodeling is on, if not off * / u (1) sps_reshaper_signal_type / * e.g.: 0: SDR, 1: PQ, 2: HLG * / u (2) sps_in_loop_filter_opt_flag / * ILFOPT flag * / u (1) sps_luma_based_chroma_qp_offset_flag / * chroma flagDQPAjustment * / u (1) sps_reshaper_active_ROI_flag u (1) if (sps_reshaper_active_ROI_flag) {reshaper active ROI in CTUsize left eu (v) reshaper_active_ROI_in_CTUsize_right eu (v) reshaper_active_ROI_in_CTUsize_top eu (v) reshaper_active_ROI_in_CTUsize_bottom eu (v) }
Petition 870200044274, of 07/04/2020, p. 77/88
71/76 sps_in_loop_filter_opt_flag equal to 1 specifies that a loop filter optimization is performed on the remodeled domain in the encoded video sequence (CVS). sps_in_loop_filter_opt _flag equal to 0 specifies that a loop filter optimization is performed in the domain not remodeled in CVS.
sps_luma_based_chroma_qp_offset_flag equal to 1 specifies that a luma-based chroma QP shift is derived (for example, according to Tables 11 or 12) and applied to the chroma encoding of each CU in the encoded video sequence (CVS). sps_luma_based_chroma_qp_offset_flag equal to 0 specifies that a chroma QP offset based on luma is not enabled in CVS.
Table 21: Example syntax for slice-level remodeling
reshaping sliceheader table model () { Described r reshape_model_profile_type eu (v) reshape_model_scale_idx u (2) reshape_model_min_bin_idx u (5) reshape_model_max_bin_idx u (5) reshape model in a band u (4) for (i = 0; i <reshape_model_num_band; i ++) {reshape_model_band_profile_delta [i] u (1) }if (sps reshaper active ROI flag) {reshape_model_ROI_modification_flag u (D if (reshape model ROI modification flag) {reshaper_ROI_mod_offsetJeft if (v) reshaper ROI mod offset right if (v) reshaper_ROI_mod_offset_top if (v) ROI reshaper mod offset bottom if (v) }}____________________________________________} _____________________________________________
Improving chroma quality [0142] Proponents of HLG-based coding argue that it provides better backward compatibility with SDR signaling.
Therefore, in theory, HLG-based signals can employ the same
Petition 870200044274, of 07/04/2020, p. 78/88
72/76 encoding configurations as legacy SDR signals. However, when viewing HLG-encoded signals in HDR mode, some color artifacts can still be observed, especially in achromatic regions (such as white and gray). In one embodiment, these artifacts can be reduced by adjusting the chromaQPOffset values during encoding. It is suggested that for a HLG content a less aggressive chromaQP setting is applied compared to that used when encoding PQ signals. For example, in Ref. [10], the model for assigning shifts from QP to Cb and Cr based on the QP of luma and a factor based on the primary colors of capture and representation is described as:
QPoffsetCb = Clip3 (-12.0, Round (c _cb * (fc * QP + /))), (18a)
QPoffsetCr = Clip3 Ç — 12.0, Round (c _cr * (fc * QP + /))), (18b) where Ccb = 1 if the primary colors of capture are equal to the primary colors of representation, c _C b = 1 , 04 if the primary colors of capture are equal to the primary colors of P3D65 and the primary colors of representation are equal to the primary colors of Rec. ITU-R BT.2020, and Ccb = 1.14 if the primary colors of capture are equal the primary colors of Rec. ITU-R BT.709 and the primary colors of representation are the same as the primary colors of Rec. ITU-R BT.2020. Similarly, c _cr = 1 if the primary colors of capture are equal to the primary colors of representation, Ccr = 1.39 if the primary colors of capture are equal to the primary colors of P3D65 and the primary colors of representation are equal to the colors primary colors of Rec. ITU-R BT.2020, and c _cr = 1.78 if the primary colors of capture are equal to the primary colors of Rec. ITU-R BT.709 and the primary colors of representation are equal to the primary colors of Rec ITU-R BT.2020. Finally, k = -0.46 and l = 0.26.
[0143] In one modality, it is proposed to use the same model, but with different parameters that produce a less aggressive chromaQPOffset change. For example, without limitation, in one modality, for Cb in equation (18a), ccb = 1, k = -0.2, and I = 7, and for Cr in equation (18b), ccr = 1, k = - 0.2, and I = 7.
Petition 870200044274, of 07/04/2020, p. 79/88
73/76
Figures 6A and 6B depict examples of how chromaQPOffset values change according to the luma quantization (QP) parameter for PQ (Rec. 709) and HLG. The values related to PQ change more dramatically than the values related to HLG. Figure 6A corresponds to Cb (equation (18a)), while Figure 6B corresponds to Cr (equation (18b)).
References [0144] Each of the references listed in this document is incorporated in its entirety as a reference.
[1] PCT application PCT / US2016 / 025082, In-Loop Block-Based Image Reshaping in High Dynamic Range Video Coding, filed on March 30, 2016, also published as WO 2016/164235, by G-M. Su.
[2] D. Bayion, Z. Gu, A. Luthra, K. Minoo, P. Yin, F. Pu, T. Lu, T. Chen, W. Husak, Y. He, L. Kerofsky, Y. Ye , B. Yi, “Response to Call for Evidence for HDR and WCG Video Coding: Arris, Dolby and InterDigital”, Doc. M36264, July (2015), Warsaw, Poland.
[3] U.S. Patent Application 15 / 410,563, Content-Adaptive Reshaping for High Codeword representation Images, filed on January 19, 2017, by T. Lu et al.
[4] PCT application PCT / US2016 / 042229, Signal Reshaping and Coding for HDR and Wide Color Gamut Signals, filed on July 14, 2016, also published as WO 2017/011636, porP. Yin etal.
[5] “Exploratory Test Model for HDR extension of HEVC”, K. Minoo et al., Document resulting from MPEG, JCTVC-W0092 (m37732), 2016, San Diego, USA.
[6] E. François, J. Sole, J. Strõm, P. Yin, “Common Test Conditions for HDR / WCG video coding experiments', JCTVC doc. Z1020, Geneva, January 2017.
[7] A. Segall, E. Francois, and D. Rusanovskyy, “JVET common test conditions and evaluation procedures for HDR / WCG Video,” JVET-E1020, ITU- Conference
Petition 870200044274, of 07/04/2020, p. 80/88
74/76
T, Geneva, January 2017.
[8] JEM software 6.0: https://ivet.hhi.fraunhofer.de/svn/svn HMJEMSoftware Ztags / HM-16.6-JEM-6.0 [9] Provisional Patent Application with US Serial Number 62 / 406.483, filed at October 11, 2016, “Adaptive Chroma Quantization in Video Coding for Multiple Color Imaging Formats,” T. Lu et al., Also filed as US Serial Patent Application 15 / 728,939, published as US Patent Application Publication 2018/0103253.
[10] J. Samuelsson et al. (Eds), “Conversion and coding practices for HDR / WCG Y’CbCr 4: 2: 0 Video with PQ Transfer Characteristics,” JCTVC-Y1017, ITU-T / ISO Conference, Chengdu, October 2016.
[11] ITU-T H.265, “High efficiency video coding,“ ITU, version 4.0, (12/2016). IMPLEMENTATION OF EXAMPLIFYING COMPUTATIONAL SYSTEM [0145] The modalities of the present invention can be implemented with a computer system, systems configured as a set of electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a programmable port arrangement in the field (FPGA), or other configurable or programmable logic device (PLD), a discrete-time or digital signal processor (DSP), an IC for specific applications (ASIC), and / or a device that includes one or more of these systems, devices or components. The computer and / or IC can carry out, control or execute instructions regarding the integrated signal reshaping and image coding, such as those described in this document. The computer and / or IC can compute any of a variety of parameters or values that refer to signal remodeling and coding processes described in this document. The image and video modes can be implemented in hardware, software, firmware and various combinations thereof.
Petition 870200044274, of 07/04/2020, p. 81/88
75/76 [0146] Certain implementations of the invention comprise computational processors that execute software instructions that induce processors to perform a method of the invention. For example, one or more processors on a screen, an encoder, a signal decoder, a transcoder or the like can implement methods related to the remodeling and encoding of integrated signals as described earlier by executing software instructions in a program memory accessible to users. processors. The invention can also be provided in the form of a program product. The program product can comprise any non-transitory medium that carries a set of computer-readable signals that comprise instructions that, when executed by a data processor, induce the data processor to execute a method of the invention. The program products according to the invention can be any of a wide variety of shapes. The program product may comprise, for example, physical media, such as magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, Flash RAM, or the like. The computer-readable signals in the program product can be optionally compressed or encrypted.
[0147] When a component (for example, a software module, processor, assembly, device, circuit, etc.) is referred to above, except where otherwise noted, the reference to that component (including a reference to a “medium ”) Should be interpreted as including equivalents of that component any component that performs the function of the described component (for example, that is functionally equivalent), including components that are not structurally equivalent to the revealed structure that performs the function in the illustrated exemplary embodiments of the invention.
Petition 870200044274, of 07/04/2020, p. 82/88
76/76
EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS [0148] Soon, exemplary modalities are described that refer to the efficient remodeling and encoding of integrated signals from images. In the previous specification, the modalities of the present invention have been described with reference to several specific details that may vary from implementation to implementation. Therefore, the unique and exclusive indicator that indicates what the invention is about, and is intended by the claimants of what the invention is, is the claim framework arising from this application, in the specific form in which these claims refer, including any subsequent correction . Any definitions expressly provided in this document for the terms contained in those claims must override the meaning of those terms used in the claims. Therefore, without limitation, elements, properties, resources, advantages or attributes that are not expressly mentioned in a claim should limit the scope of that claim in any way. Correspondingly, the specification and drawings should be considered in an illustrative rather than a restrictive sense.

权利要求:
Claims (10)
[1]
1. Method for encoding images with a processor, the method CHARACTERIZED by the fact that it comprises:
accessing an input image (117) in a first code word representation with a processor;
generate a progressive reshaping function that maps pixels from the input image to a second code word representation, in which the second code word representation allows for more efficient compression than the first code word representation;
generate an inverse reshape function based on the progressive reshape function, in which the inverse reshape function maps pixels from the second encoding representation to the first encoding representation;
for an input pixel region in the input image;
compute (225) a predicted region based on pixel data in a reference frame buffer or previously encoded spatial neighbors;
generate a remodeled residual region based on the input pixel region, on the predicted region, and on the progressive remodeling function, where generating the remodeled residual region comprises computing:
Res_r (/) = a (Pred_sample (i)) * (Orig_sample (i) -Pred_sample (i)), where Res_r (i) denotes a pixel of the remodeled residual region, Pred_sample (j) denotes a pixel of the predicted region, a (Pred_sample (i)) denotes a scaling factor as a function of the value of Pred_sample (j), where a (J denotes the progressive reshaping function, and Orig_sample (í) denotes a pixel of the input image region;
generate a quantized remodeled residual region based on the remodeled residual region;
Petition 870200044274, of 07/04/2020, p. 84/88
[2]
2/4 generate a deconstructed residual remodeled region based on the quantized remodeled residual region;
to generate a reconstructed pixel region based on the deconstructed residual remodeled region, in the predicted region, and in the inverse remodeling function, in which the reconstructed pixel region consists of computing:
Reco_sample (/) = Pred_sample (i) + (Ma (Pred_sample (i))) * Res_d (i), where Reco_sample (/) denotes a pixel of a reconstruction pixel region, Res_d (j) denotes a pixel of the region deconstructed residual remodeled, and 1 / aQ denotes the inverse remodeling function; and generating a reference pixel region to be stored in the reference frame buffer based on the reconstructed pixel region.
2. Method, according to claim 1, CHARACTERIZED by the fact that it further comprises:
generate a remodel signaling bit stream that characterizes the progressive remodeling function and / or the reverse remodeling function; and multiplexing the reshaper bit stream with an encoded bit stream generated based on the input image to generate an output bit stream.
[3]
3. Method, according to claim 1, CHARACTERIZED by the fact that generating the quantized remodeled residual region comprises:
apply a progressive coding transform to the remodeled residual region to generate transformed data; and apply a progressive coding quantizer to the transformed data to generate quantized data.
[4]
4. Method, according to claim 3, CHARACTERIZED by the fact that it will generate deconstructed residual remodeled region comprises:
applying a reverse encoding quantizer to the reshaped quantized residual region to generate inverse quantized data; and
Petition 870200044274, of 07/04/2020, p. 85/88
3/4 apply an inverse coding transform to the inverse quantized data to generate the deconstructed residual remodeled region.
[5]
5. Method, according to claim 1, CHARACTERIZED by the fact that generating the reference pixel region to be stored in the reference frame buffer comprises applying a loop filter to the reconstructed pixel region.
[6]
6. Method according to claim 1, CHARACTERIZED by the fact that the input pixel region comprises an image of the region of interest.
[7]
7. Method for decoding a coded bit stream with a processor to generate an output image in a first codeword representation, the CHARACTERIZED method because it comprises:
receive an encoded image;
receive remodeling information for the encoded image;
generate, based on the remodeling information, a progressive remodeling function that maps pixels from the first codeword representation to the second codeword representation, where the second codeword representation allows for more efficient compression than the first code word representation, in which the progressive reshaping function consists of a reshaping scaling function a () based on pixel values;
generate, based on the remodeling information, an inverse remodeling function, in which the inverse remodeling function maps pixels from the second codeword representation to the first codeword representation, in which the inverse remodeling function is 1 / a ();
for a region of the encoded image;
generate a reshaped decoded residual region;
generate a predicted region based on pixels in a reference pixel buffer or previously decoded spatial neighbors;
Petition 870200044274, of 07/04/2020, p. 86/88
4/4 generate a reconstructed pixel region based on the decoded residual remodeled region, in the predicted region, and on the reverse remodeling function, where generating the reconstructed pixel region comprises computing:
Reco_sample (/) = Pred_sample (j) + (1 la (Pred_sample (i))) * Res_d (j), where Reco_sample (/) denotes a pixel of a reconstruction pixel region, Res_d (i) denotes a pixel of decoded remodeled residual region, (Pred_sample (i)) denotes a remodeling scaling factor as a function of the value of Pred_sample (j), and Pred_sample (j) denotes a pixel of the predicted region; and generating an output pixel region for the output image based on the reconstructed pixel region; and store the output pixel region in the reference pixel buffer.
[8]
8. Method according to claim 7, CHARACTERIZED by the fact that the region of the encoded image comprises an image of region of interest.
[9]
9. Method, according to any one of claims 1 to 7, CHARACTERIZED by the fact that it also comprises methods to optimize decisions related to coding based on the progressive remodeling function, in which decisions related to coding comprise one or more decisions inter / intra mode, dQP optimizations, rate distortion optimizations, cross-component linear model prediction, residual prediction, adaptive clipping, or loop filtering.
[10]
10. Non-transient computer-readable storage medium, CHARACTERIZED by the fact that computer-executable instructions are stored in it to execute a method as defined in any of claims 1 to 9.

类似技术:

---

公开号 | 公开日 | 专利标题

---

BR112019016885A2|2020-04-14|integrated image reshaping and video encoding

---

AU2018203223B2|2020-01-16|Chroma quantization in video coding

---

JP2018514145A|2018-05-31|In-loop block-based image reconstruction for high dynamic range video coding

---

BR112020006985A2|2020-10-06|video encoding with spatially variable quantization adaptable to content

---

US20210211738A1|2021-07-08|Image reshaping in video coding using rate distortion optimization

---

CN110612724B|2021-04-20|Quantization parameter prediction using luma information

---

US20210029361A1|2021-01-28|Piece-wise-linear mapping for in-loop reshaping in image coding

---

JP7024102B2|2022-02-22|Image shaping in video coding with rate-distortion optimization

---

TW202046732A|2020-12-16|Method and apparatus of video coding

同族专利:

---

公开号 | 公开日

---

RU2020122372A3|2021-02-17|

---

WO2019006300A1|2019-01-03|

---

US20200267392A1|2020-08-20|

---

JP2020526942A|2020-08-31|

---

RU2020122372A|2020-09-24|

---

RU2727100C1|2020-07-17|

---

KR20200021913A|2020-03-02|

---

EP3571838A1|2019-11-27|

---

RU2746981C2|2021-04-22|

---

CN110301134A|2019-10-01|

---

US10992941B2|2021-04-27|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

---

KR100261253B1|1997-04-02|2000-07-01|윤종용|Scalable audio encoder/decoder and audio encoding/decoding method|

---

FI117845B|2004-08-12|2007-03-15|Gurulogic Microsystems Oy|Video image processing|

---

US7876833B2|2005-04-11|2011-01-25|Sharp Laboratories Of America, Inc.|Method and apparatus for adaptive up-scaling for spatially scalable coding|

---

US8437053B2|2010-04-15|2013-05-07|Eastman Kodak Company|Gamut mapping using hue-preserving color space|

---

US9161041B2|2011-01-09|2015-10-13|Mediatek Inc.|Apparatus and method of efficient sample adaptive offset|

---

CN104054338B|2011-03-10|2019-04-05|杜比实验室特许公司|Locating depth and color scalable video|

---

US8334911B2|2011-04-15|2012-12-18|Dolby Laboratories Licensing Corporation|Encoding, decoding, and representing high dynamic range images|

---

US9036042B2|2011-04-15|2015-05-19|Dolby Laboratories Licensing Corporation|Encoding, decoding, and representing high dynamic range images|

---

JP5601333B2|2012-02-09|2014-10-08|コニカミノルタ株式会社|Image processing apparatus, threshold matrix shaping method and program|

---

CN105324997B|2013-06-17|2018-06-29|杜比实验室特许公司|For enhancing the adaptive shaping of the hierarchical coding of dynamic range signal|

---

EP3198556B1|2014-09-26|2018-05-16|Dolby Laboratories Licensing Corp.|Encoding and decoding perceptually-quantized video content|

---

US10136133B2|2014-11-11|2018-11-20|Dolby Laboratories Licensing Corporation|Rate control adaptation for high-dynamic range images|

---

WO2016154336A1|2015-03-25|2016-09-29|Dolby Laboratories Licensing Corporation|Chroma subsampling and gamut reshaping|

---

EP3281409B1|2015-04-06|2019-05-01|Dolby Laboratories Licensing Corporation|In-loop block-based image reshaping in high dynamic range video coding|

---

WO2016172091A1|2015-04-22|2016-10-27|Dolby Laboratories Licensing Corporation|Signal reshaping and coding in the ipt-pq color space|

---

WO2017011636A1|2015-07-16|2017-01-19|Dolby Laboratories Licensing Corporation|Signal reshaping and coding for hdr and wide color gamut signals|

---

US10165275B2|2016-01-26|2018-12-25|Dolby Laboratories Licensing Corporation|Content-adaptive reshaping for high dynamic range images|

---

US10477212B2|2016-10-11|2019-11-12|Dolby Laboratories Licensing Corporation|Adaptive chroma quantization in video coding for multiple color imaging formats|

---

US10757428B2|2018-10-10|2020-08-25|Apple Inc.|Luma and chroma reshaping of HDR video encoding|US10728549B2|2017-11-06|2020-07-28|Dolby Laboratories Licensing Corporation|Adaptive loop filtering for high-dynamic range video|

---

KR20210089132A|2018-11-06|2021-07-15|베이징 바이트댄스 네트워크 테크놀로지 컴퍼니, 리미티드|Intra prediction based on location|

---

AU2019391197A1|2018-12-07|2021-06-17|Beijing Bytedance Network Technology Co., Ltd.|Context-based intra prediction|

---

WO2020142186A1|2019-01-04|2020-07-09|Interdigital Vc Holdings, Inc.|Inverse mapping simplification|

---

WO2020156532A1|2019-02-01|2020-08-06|Beijing Bytedance Network Technology Co., Ltd.|Restrictions on in-loop reshaping|

---

EP3903482A1|2019-02-22|2021-11-03|Beijing Bytedance Network Technology Co. Ltd.|Neighbouring sample selection for intra prediction|

---

EP3700217A1|2019-02-22|2020-08-26|InterDigital VC Holdings, Inc.|Inverse mapping simplification|

---

EP3903493A1|2019-02-24|2021-11-03|Beijing Bytedance Network Technology Co. Ltd.|Parameter derivation for intra prediction|

---

AU2020229608A1|2019-02-28|2021-10-07|Lg Electronics, Inc.|APS signaling-based video or image coding|

---

CN113519157A|2019-03-04|2021-10-19|北京字节跳动网络技术有限公司|Two-stage signaling of filtering information in video processing|

---

WO2020180120A1|2019-03-05|2020-09-10|엘지전자 주식회사|Method and device for image coding based on lmcs|

---

KR20210118194A|2019-03-07|2021-09-29|엘지전자 주식회사|Video or video coding based on luma mapping and chroma scaling|

---

CN113545070A|2019-03-08|2021-10-22|北京字节跳动网络技术有限公司|Signaling of shaping information in video processing|

---

WO2020184928A1|2019-03-11|2020-09-17|엘지전자 주식회사|Luma mapping- and chroma scaling-based video or image coding|

---

CN113632469A|2019-03-23|2021-11-09|北京字节跳动网络技术有限公司|Default in-loop shaping parameters|

---

CN113711590A|2019-04-18|2021-11-26|北京字节跳动网络技术有限公司|Reducing delay in cross component mode|

---

CN114026866A|2019-05-03|2022-02-08|交互数字Vc控股公司|Chroma processing for video encoding and decoding|

---

WO2020231139A1|2019-05-16|2020-11-19|엘지전자 주식회사|Luma mapping- and chroma scaling-based video or image coding|

---

WO2020256510A1|2019-06-21|2020-12-24|현대자동차주식회사|Method and device for controlling coding tools|

---

WO2020253861A1|2019-06-21|2020-12-24|Beijing Bytedance Network Technology Co., Ltd.|Adaptive in-loop color-space transform for video coding|

---

WO2020254564A1|2019-06-21|2020-12-24|Interdigital Vc Holdings France, Sas|Luma mapping with chroma scalinglut extension and clipping|

---

WO2020262915A1|2019-06-24|2020-12-30|엘지전자 주식회사|Video or image coding using alf or lmcs|

---

US11172237B2|2019-09-11|2021-11-09|Dolby Laboratories Licensing Corporation|Inter-layer dynamic range scalability for HDR video|

---

WO2021195569A1|2020-03-27|2021-09-30|Beijing Dajia Internet Information Technology Co., Ltd.|Methods and devices for prediction dependent residual scaling for video coding|

---

WO2021222871A1|2020-04-30|2021-11-04|Beijing Dajia Internet Information Technology Co., Ltd.|Methods and devices for prediction dependent residual scaling for video coding|

---

WO2021263029A1|2020-06-24|2021-12-30|Qualcomm Incorporated|Model parameter derivation of local illumination compensation in the luma mapping with chroma scaling-mapped domain in video coding|

法律状态:
2021-10-19| B350| Update of information on the portal [chapter 15.35 patent gazette]|

优先权:

---

申请号 | 申请日 | 专利标题

---

US201762526577P| true| 2017-06-29|2017-06-29|

---

US201762561561P| true| 2017-09-21|2017-09-21|

---

US201862629313P| true| 2018-02-12|2018-02-12|

---

US201862680710P| true| 2018-06-05|2018-06-05|

---

US201862686738P| true| 2018-06-19|2018-06-19|

---

PCT/US2018/040287|WO2019006300A1|2017-06-29|2018-06-29|Integrated image reshaping and video coding|

---